USING FINDINGS OF AOL FOR QUALITY ENHANCEMENT FOR THE
VII. A. Quality of the Academic Program Improvement Plan
There are several layers in the continuous improvement cycle approach to AOL assessment adopted by the Coles College of Business. In this section, we examine plans for improvement at the following levels:
• Within courses, e.g. topical coverage, instructional methods, grade evaluation processes.
• In the course-embedded assessment process itself, e.g. assessment vehicles, sampling process, benchmarking standards.
• In the indirect program assessment methods, e.g. exit surveys, employer surveys.
According to our AOL process, Course Assessment Teams (CATs) are expected to collect, analyze, and review the results of their assessment activities. Following the review of results, for each course evaluated, the CAT should complete a “Description of Planned Changes in Curricula, Pedagogy, Teaching and Learning Materials and / or Learning Goals and Objectives” which answers the following questions:
• What did you learn?
• What is the Department going to do about it (how)?
• Department plans for the next academic year?
Courses assessed during the 2005 year included:
Spring 2005: BISM 2100, ECON 2300, MGT 3200, Fall 2005: FIN 3100, MGT 3100, MKTG 3100
Plans for improvement for each of these six business Core courses are presented on the next several pages in Exhibits VI-1 through VI-6.
SACS AOL Report – Spring 2006 – Department of Accounting – 111 Students to Meet
or Exceed Meet or Exceed
Expectations
Description of Planned Changes in Curricula, Pedagogy, Teaching and Learning Materials and / or Learning Goals and Objectives:
What did we learn?
Four learning objectives were assessed in each of three course sections for a total of 12 measurements. The actual performance on all twelve measurements met or exceeded the benchmark prediction of performance. On one of the 12 measurements (6.1 in Section 3), the result just met the minimum predicted performance. The course coordinator and the instructor involved in the assessment believe that there were problems with the design of one of the assessment vehicles and that it may not accurately measure the learning outcome.
What are we going to do about it (how)?
The course coordinator proposed the following changes to the course and/or assessment vehicles as a result of the assessment experience:
Select a new text that incorporates activities for both MS Office application and communication skill development.
Restate course objectives to more clearly focus on learning objectives.
SACS AOL Report – Spring 2006 – Department of Accounting – 112
Adopt an assessment vehicle – Train and Assess Information Technology [TAIT] – that generates more consistent student assignments and allows instructors to better determine student performance.
Develop a customized WebCT course template for course structure.
Departmental Action Plan based on Results of the Course-Embedded Assessment Process – The Department AOL Committee accepted the recommendations of the course coordinator. The course coordinator made changes in the course based on the spring of 2005 assessment experience.
A new text and related course revisions were launched in the fall semester of 2005. The course is being assessed again this semester (Spring 2006) per the AOL Council’s timetable. A new assessment vehicle (TAIT) is being used this semester (Spring 2006).
In order to improve the Department AOL Committee’s understanding and coordination of the AOL process for the BISM courses, a BISM faculty member (the course coordinator for BISM 2100) was added to the AOL committee beginning Fall 2005.
SACS AOL Report – Spring 2006 – Department of Accounting – 113
Exhibit VII-2: ECON 2300 – Action Plan Learning
Objective Assessed
Current % of Students to Meet
or Exceed Expectations
Current Benchmarking
Standard
% of Students to Meet or Exceed Expectations after
changes have been Implemented
New Benchmarking
Standard (if applicable)
Learning Objective 1.2
69.7% 80%
Learning Objective 1.3
77.7% 70%
Learning Objective 6.1
79.5% 80%
Description of Planned Changes in Curricula, Pedagogy, Teaching and Learning Materials and / or Learning Goals and Objectives:
What did we find out?
Performance outcomes in this course were quite mixed. The minimum benchmarking standard was exceeded for learning objective 1.3, almost met for learning objective 6.1 and not met for learning objective 1.2
What are we going to do about it (how)?
The departmental AOL Committee concluded that rather than makes changes to the course, the CAT for ECON 2300 should do the following:
• Consider the Current Benchmarking Standards to determine whether or not they are appropriate for this level of quantitative course.
• Evaluate the assessment vehicles used to determine appropriateness and effectiveness.
• Administer the assessment in multiple sections covered by different faculty during different times of the day.
Departmental Action Plan based on Results of the Course-Embedded Assessment Process – This course will be assessed again in the fall of 2006 at which time the above actions will be taken.
In the meantime, efforts are underway to develop a common assessment tool for all sections and to increase awareness of the AOL process among all faculty.
SACS AOL Report – Spring 2006 – Department of Accounting – 114
Exhibit VII-3: FIN 3100 – Action Plan
Learning Objective Assessed
Current % of Students to Meet
or Exceed Expectations
Current Benchmarking
Standard
% of Students to Meet or Exceed Expectations after
changes have been Implemented
New Benchmarking
Standard (if applicable)
Learning Objective 1.3
73.4% 60%
Learning Objective 4.1
74.7% 60%
Learning Objective 4.2
86.1% 60%
Learning
Objective 7.4.2 72.7% 60%
Description of Planned Changes in Curricula, Pedagogy, Teaching and Learning Materials and / or Learning Goals and Objectives:
What did we find out?
Performance outcomes exceeded the current benchmarking standards for all learning objectives. In one section, the instructor failed to collect assessment measures for two of the learning objectives, 1.3 and 7.4.2. The AOL Committee learned a great deal about the assessment process itself in this initial implementation; more structured guidelines and reporting templates have been developed as a result.
Departmental Action Plan based on Results of the Course-Embedded Assessment Process – The results of the assessment processes for FIN 3100 and ECON 8010 were discussed in a
department meeting on August 18, 2005. The following conclusions were derived:
• The benchmarking standard for the % of students that meet or exceed expectations for FIN 3100 of 60% may be too low.
• The benchmarking standard for the % of students that meet or exceed expectations for ECON 8010 of 90% may be too high.
• As a department, we need to determine an ideal range for benchmarking standards used in the assessment of graduate level and undergraduate level courses.
• As a department, we need to develop continuous improvement goals for the assessment process. One area of concern is how to address small sample size issues and the
consequences of Type I errors. It is difficult to determine if changes are indeed necessary based on the assessment results from a sample of 34 students. We may need to develop trend lines over time or increase our sample sizes.
Department Plans for the 2005/06 Academic Year:
• Administrative Changes in the Assessment Process: The departmental AOL Committee will develop a detailed checklist to accompany the report templates faculty need to
complete during the assessment process. This should reduce much of the confusion which led to incomplete reports and files during the past spring term.
SACS AOL Report – Spring 2006 – Department of Accounting – 115
• Benchmarking Standards: The departmental AOL Committee will lead discussion necessary to determine realistic benchmarking standards.
• Continuous Improvement Goals: The department AOL Committee will lead discussion necessary to develop continuous improvement goals for the department related to student mastery of learning objectives.
SACS AOL Report – Spring 2006 – Department of Accounting – 116
Exhibit VII-4: MGT 3100 – Action Plan
Learning Objective Assessed
Current % of Students to Meet
or Exceed Expectations
Current Benchmarking
Standard
% of Students to Meet or Exceed Expectations after
changes have been Implemented
New Benchmarking
Standard (if applicable)
Learning Objective 1.1
86% 70-80%
Learning Objective 2.1
94% 70-80%
Learning Objective 2.2
94% 70-80%
Learning Objective 3.1
84% 70-80%
Learning Objective 3.2
93% 70-80%
Learning Objective 3.3
82% 70-80%
Learning Objective 3.4
92% 70-80%
Learning Objective 4.1
88% 70-80%
Learning Objective 4.2
88% 70-80%
Description of Planned Changes in Curricula, Pedagogy, Teaching and Learning Materials and / or Learning Goals and Objectives:
Description of Planned Changes in Curricula, Pedagogy, Teaching and Learning Materials and / or Learning Goals and Objectives:
MGT 3100: MANAGEMENT AND BEHAVIORAL SCIENCE