Program- and Course-Level Assessment

(1)

Wharton County Junior College

Program- and Course-Level

Assessment

The Do’s and the PLEASE Do Not’s

Presented by Amanda Allen, Ed.D.

(2)

The Purpose of Assessment

Continual planning and assessment are critical to maintaining and improving upon the quality of instruction.

Moreover, the institution uses instructional assessment data to monitor student success.

By engaging in continuous and purposeful assessment practices, programs will be able to readily identify areas of strength and weakness, use data to inform decisions related to the modification of curriculum, and improve upon desired outcomes.

(3)

SACSCOC Standard

And…it’s required by SACSCOC.

Section 8.2:

The institution identifies expected outcomes, assesses the extent to which it achieves these outcomes, and provides evidence of seeking improvement based on analysis of the results in Student Learning Outcomes for each of it’s educational programs.

(4)

Building Blocks of Assessment

Breaking down the SACSCOC requirements, there are three items we have to attend to:

Outcomes: What do we expect students to learn?

Assessment of those outcomes: Did students achieve

what we wanted them to?

Evidence of seeking improvement: Based on student

performance, what are you going to change to make our curriculum better?

(5)

Program Learning Outcomes

Each Degree or Certificate Program (AA in English; AAS in Manufacturing Technology) must have a set of Program Learning Outcomes (PLOs). This means program, not department. If your department has areas that don’t result in the awarding of a degree or certificate, we do not need PLOs for that area.

Most programs should have 4 – 8 PLOs.

PLOs are the overarching objectives that answer the question, “What are the 4 – 8 big‐picture competencies that a student should know/be able to do when they graduate with this degree?”

Programs that are accredited should use the PLOs that are designated by your professional accrediting agency for your program.

(6)

Program Learning Outcomes (Examples) Business Administration • Students will be able to effectively communicate, both through oral and written means, with professionals in their field. • Students will apply foundational principles of financial and managerial accounting. • Students will understand an analysis of the economy as a whole and analysis of the behavior of economic markets. • Students will develop in‐depth knowledge of commonly used office software, specifically Microsoft Office.

(7)

Program Learning Outcomes (Examples) Speech • Students will examine the modes of communication and settings in which communications occurs. • Students will examine how people in various social contexts interact. • Students will develop skills which will facilitate effectiveness in the workplace. • Students will examine how communication skills are acquired. • Students will learn how communication shapes societies and cultures.

(8)

Program Learning Outcomes (Examples) EMS • Demonstrate the ability to systematically collect, analyze and record data on patient health status using methods consistent with recognized national standards. • Acquire and synthesize information in a critical, scientific and effective manner. • Discern and manage ethical issues in a professional manner. • Provide quality patient treatment under emergency conditions in a timely and effective manner that is consistent with nationally recognized standards. • Communicate professionally and effectively within all aspects of the healthcare environment. • Provide skilled emergency care using the highest professional knowledge, judgment, ethics and behavior.

(9)

PLOs must be measureable and focused on student learning.

Some commonly used PLOs that are not actually PLOs and should be avoided are:

• Students will be able to transfer to a university. • Students will be able to get an entry‐level job. • Student enrollment/completion will increase.

• Program budget will be increased by 10% to pay for new instructional materials.

(10)

At WCJC, we use our pre‐existing course assessment process to do “double duty”, allowing for simultaneous assessment of our Course‐Level SLOs and Program‐ Level Outcomes using the PCLAR template.

The PCLAR includes a column in which to note the Program‐Level Outcome associated with each Course‐ Level SLOs. This column MUST be completed.

Use the newest version of the PCLAR template (posted on the “Forms” page on the intranet) for the current and all future semesters.

(11)

FAQs

I have my own system for monitoring Course‐Level SLO assessment. Can I turn that in to you instead of the PCLAR?

No.

But I really like my tracking system.

That’s not a question…but no.

Many of you may have your own system for gathering Course‐Level SLO data. That’s fine! However, when it’s time to submit your annual report, that submission must use the PCLAR template.

(12)

FAQs

How many PLOs can I have associated with each Course‐Level SLO?

Since Course‐Level SLOs are very specific (compared to PLOs), they should only link to a single Program‐ Learning Outcome.

In some cases, they may link to a couple of PLOs.

NO Course‐Level SLO should be linked to more than 2

PLOs. The second column on your PCLAR should not list all of your PLOs for every Course‐Level SLO.

(13)

FAQ Example This…. Course SLO (# and text) PLO # Students will demonstrate… 5 Students will compare…. 2 Students will explain… 3 Students will recognize… 1, 2 Not This…. Course SLO (# and text) PLO # Students will demonstrate… 1,2,3,4,5 Students will compare…. 1,2,3,4,5 Students will explain… 1,2,3,4,5

(14)

FAQs

Do all PLOs have to be listed/referenced in each course?

No. Some courses may include content that relates to every PLO. Other courses may only provide content over a few.

However, every PLO must be represented several times on your PCLAR.

If you have a Program‐Level Outcome that is not taught in the majority of your major classes, it must not be that important and should be reconsidered.

(15)

Breaking down the SACSCOC requirements, there are three items to attend to:

(16)

Outcomes Assessment

Outcome assessment includes several key items:

• A metric which is used to measure student achievement and the means through which it will be assessed.

• A criteria for success (target) which indicates the level of achievement that must be reached in order to consider students to be successful in learning what was required.

• A timeline for measurement. Since we use our course‐level assessments, our assessment timelines are understood to be every three years.

(17)

Outcomes Assessment (Example)

Course‐Level SLO:

Demonstrate accurate calculation of compound interest.

Program‐Level Outcome:

PLO2: Students will apply foundational principles of financial and managerial accounting.

Metric/Means of Assessment:

Final exam question #14.

Criteria for Success:

85% of students will answer this exam question correctly.

Timeline:

(18)

Research, develop and deliver extemporaneous speeches with effective verbal and nonverbal techniques.

PLO 3: Students will develop skills which will facilitate effectiveness in the workplace.

Delivery/body sections of persuasive rubric.

80% of students will score 3 or better on this rubric item.

Timeline:

(19)

Differentiate the various types of health care facilities and their records.

PLO 1: Demonstrate the ability to systematically collect, analyze, and record data on patient health status using methods consistent with recognized national standards.

Provider's Roles Assignment

75% of students will achieve a grade of 75% or above on the assignment.

Timeline:

(20)

Improving Outcomes Assessment

Can you spot the (very common) mistakes?

Course SLO Means of

Assessment Criteria for Success Students will demonstrate… Final Exam 75% of students will pass the

final exam.

Students will compare…. Final Exam 75% of students will pass the final exam.

Students will explain… Final Exam 75% of students will pass the final exam.

Students will analyze… Group Presentation

75% of students will make a “C” or better on the group presentation.

(21)

Course SLO Means of Assessment Criteria for Success Students will demonstrate… Final Exam 75% of students will pass the final exam. Students will compare…. Final Exam 75% of students will pass the final exam. Students will explain… Final Exam 75% of students will pass the final exam.

Think beyond the exam! Exams are readily available assessment tools, but there

are many other ways that you assess student learning…projects, presentations, papers, group assignments, etc. Your assessment report should not rely solely on exam questions. You can also use students’ self‐reported understanding using in‐ class surveys or course evaluations, but this should be limited.

(22)

Assessment Criteria for Success Students will demonstrate… Final Exam 75% of students will pass the

final exam.

Students will compare…. Final Exam 75% of students will pass the final exam.

Students will explain… Final Exam 75% of students will pass the final exam.

Final exam/project grades are not specific enough! When evaluating student

achievement for an SLO, identify which question(s) on the exam actually relate to that specific outcome. Use only that item(s) to determine student achievement. In addition, if your means of assessment is a project or presentation, using a rubric will allow you to focus on key learning outcomes inherent within that assignment.

(23)

Here’s an improved version:

Course SLO Means of Assessment Criteria for Success Students will demonstrate… Term Paper #1; Rubric Item #5 85% of students will score a 3 or higher on this item. Students will compare…. Final Exam Question #10 80% of students will answer this question correctly. Students will explain… Group Presentation; Rubric Item #7 85% of students will “meet expectations” on this item.

Students will recognize… In‐class survey; Item #9 90% of students will state that they have a “good” or “strong” understanding on this item of the in‐class student‐learning survey.

(24)

Outcomes Assessment

As a final step for your outcome assessment, you must collect the data for your assessment metrics, comparing that to the criteria that was set. This should be collated for all sections and entered as your “Summary of Findings”.

Assessment Criteria for Success Summary of Findings Students will demonstrate… Term Paper #1; Rubric Item #5 85% of students will score a 3 or higher on this item. 73% of students scored ≥3 on this item. Students will compare…. Final Exam Question #10 80% of students will answer this question correctly. 91% of students answered this item correctly. Students will explain… Group Presentation; Rubric Item #7 85% of students will “meet expectations” on this item. 87% of students scored “meet expectations” (or better) on this item.

(25)

Breaking down the SACSCOC requirements, there are three items to attend to:

(26)

Seeking Improvement

In the current standards, SACSCOC asks for “evidence of seeking improvement”.

Simply put, you need to include verbiage that indicates what you plan to do to improve your curriculum and student learning based on the data that you collected as part of your outcomes assessment.

This is the most important aspect of the assessment process!

(27)

Course SLO Metric Criteria for Success Summary of Findings Activities as a Result of Assessment Students will demonstrate… Term Paper #1; Rubric Item #5 85% of students will score a 3 or higher on this item. 93% of students scored ≥3 on this item. Target met. Continue to monitor. Students will compare…. Final Exam Question #10 80% of students will answer this question correctly. 91% of students answered this item correctly. Target met. Continue to monitor. Students will explain… Group Presentation; Rubric Item #7 85% of students will “meet expectations” on this item. 87% of students scored “meet expectations” (or better) on this item. Target met. Continue to monitor.

(28)

Course SLO Metric Criteria for Success Summary of Findings Activities as a Result of Assessment Students will demonstrate… Term Paper #1; Rubric Item #5 85% of students will score a 3 or higher on this item. 93% of students scored ≥3 on this item. Target met. Continue to monitor. Students will compare…. Final Exam Question #10 80% of students will answer this question correctly. 91% of students answered this item correctly. Target met. Continue to monitor. Students will explain… Group Presentation; Rubric Item #7 85% of students will “meet expectations” on this item. 87% of students scored “meet expectations” (or better) on this item. Target met. Continue to monitor.

Although all criteria for success were met/exceeded, there should still be

changes made to the curriculum in an attempt to better meet student need

and enhance student success. We know that you are continually making improvements…WRITE IT DOWN!!!

(29)

Course SLO Metric Criteria for Success Summary of Findings Activities as a Result of Assessment Students will demonstrate… Term Paper #1; Rubric Item #5 85% of students will score a 3 or higher on this item. 73% of students scored ≥3 on this item. Change the rubric item description. Students will compare…. Final Exam Question #10 80% of students will answer this question correctly. 67% of students answered this item correctly. Change the test question to clarify information being sought. Students will explain… Group Presentation; Rubric Item #7 85% of students will “meet expectations” on this item. 62% of students scored “meet expectations” (or better) on this item. Add more to lecture about this topic.

(30)

The activities that are planned do not relate to the curriculum; rather, they are focused on grading parameters, test questions, etc. While these may need revision, this is not sufficient in terms of identifying meaningful change to instruction, curriculum, and/or student learning.

(31)

This activity is not specific enough. What content will be added? Which lecture?

(32)

Here’s an improved version.

Course SLO Metric Criteria for Success Summary of Findings Activities as a Result of Assessment Students will demonstrate… Term Paper #1; Rubric Item #5 85% of students will score a 3 or higher on this item. 93% of students scored ≥3 on this item. Add an additional reading requirement from chapter 5 of the text related to appropriate use of citations. Students will compare…. Final Exam Question #10 80% of students will answer this question correctly. 91% of students answered this item correctly. Target met. Continue to monitor. Students will explain… Group Presentation; Rubric Item #7 85% of students will “meet expectations” on this item. 73% of students scored “meet expectations” (or better) on this item. Incorporate a team communication assignment to enhance group interactions and provide additional oral communication practice.

(33)

FAQs

How many activities do I have to include in my report?

For every criteria for success where the target is not met, you need to include an activity seeking improvement.

If you have met all of your targets, you need to still incorporate some actions seeking improvements. Not for every criteria, but at least every one out of every three/four.

We know that you make changes all the time to improve your curriculum; here is the place to write it down!

Also, you may want to reevaluate your targets if you are consistently meeting 100% of your targets 100% of the time…

(34)

FAQs

If the criteria of success are not met, can I decrease the target the next time the course is assessed?

The targets that you set must be determined based on what you is the minimum level for ensuring that students are successful in achieving the learning outcomes identified.

If students consistently do not hit the target that has been set, the activities identified should relate specifically to curricular changes that will enhance student learning, thereby enabling students to reach that level of achievement.

Changing a target from 75% to 70% attainment is not an appropriate activity to list on your PCLAR.

(35)

FAQs

What if I have more questions or need help?

Reach out!! It is MUCH easier to be proactive on assessment; contact me if you have questions or concerns.

Do you have any helpful resources?

YES!! Check out the Planning and IE webpage for links and other useful information!

(36)

Questions or Concerns?

Amanda A. Allen, Ed.D.

VP of Planning and Institutional Effectiveness