MASTER OF ACCOUNTING PROGRAM
VII. A. Lessons Learned So Far
The initial collection of AOL assessment data for the MAcc Program is occurring this semester (Spring 2006). Thus, we have no assessment data to use yet. Nevertheless, we have learned a great deal from the AOL process to this point and from the assessment activities we have completed for our undergraduate and MBA courses.
Most everything takes longer and/or is more complicated than expected.
The devil is in the details. It was not too difficult to get the faculty to agree on the Learning Goals and their mapping to the Core Requirements courses. Who wouldn’t agree that Critical Thinking is an appropriate Learning Goal for an Accounting program? And who teaches a graduate class that does not require Critical Thinking? However, moving to the next level of detail – the assessment vehicles and their “grading” – generated significantly more controversy. Saying that Critical Thinking is required in a course is a lot easier than actually measuring it. And if a course is taught by more than one faculty member, agreement on how to measure it is not automatic.
Don’t try to do too much. We initially mapped our undergraduate courses to all the Learning Goals we felt were addressed in each course, without regard to difficult measurement issues or resource limitations. However, when faced with identifying specific assessment vehicles and rubrics for “grading” those vehicles, we quickly realized how much work it would be to measure all the Objectives to which we had mapped each of our courses. The AACSB Accreditation Review Team that visited us in 2004 agreed and stated that we were trying to do too much. At that point we reviewed the mappings and cut the number of Objectives for each course back to three to five in an attempt to make the process more manageable. We are still having trouble getting all the work done.
The Learning Goals/Objectives mapped to a course should be significant. In one undergraduate course, the vehicles and rubrics were developed and applied with little trouble, and student performance on all the assessments was acceptable. However, the instructor recommended that one of the Learning Objectives be “un-mapped” to the course because the assignment used as the assessment vehicle was worth only 5% of the course points, indicating that it was not one of her major objectives for student learning. When we originally mapped the Objective to that course, we were thinking more about the ease of measurement rather than its relative importance in this course.
Existing evaluation methods may need to be modified. In some cases, faculty believed that assessing an Objective would be easy to do with assignments/activities already in their courses. For example, one faculty member had an Excel workbook project in her course and she planned to use it as the vehicle to assess our undergraduate Technology
Learning Goal. When the performance on the Goal did not meet expectations, she realized that part of the problem was with the “grading” of the vehicle. She had graded
the Excel workbooks for errors, as always. But some of the errors were caused by the
students’ lack of learning of Accounting content, and not their lack of learning of the use of the technology. For example, some students wrote the wrong formula to calculate a ratio because they didn’t know how the ratio should be calculated, not because they couldn’t correctly write a formula in Excel. For the next round of assessments in that course, the instructor is going to calculate scores on the Accounting content issues separately from the Excel issues.
At the beginning of the process, there is a lot to do and a lot of information to keep organized. Essentially nothing in the AOL process is ever “done.” As we are trying to initiate assessments in our courses, we are finding that the mappings need to be revised, the descriptions of the assessment vehicles already identified need to be tweaked, and so forth, based on the continual discussion during the process. In addition, the results from the first assessments and the experience of implementing them indicate changes to be made. Just keeping track of the latest versions of the mappings is a challenge. Plus, new faculty are being drawn into the process as we add new courses to the assessment
schedule and they don’t have the history with the process that the Committee members do, so they need extra guidance. Providing them with enough information without overwhelming them or confusing them is a balancing act.
Faculty attitude is extremely important. Assessment requires work. As mentioned earlier, an existing course activity may be an appropriate vehicle for AOL assessment, but even that might lead to extra work for the faculty member if the grading must be modified. In other cases, additional activities may need to be incorporated in order to properly assess our Learning Goals and Objectives. If not presented effectively, the AOL process can be seen by faculty as a lot of extra work for unknown benefit. The
Department AOL Committee is working on developing guidelines and forms to make the assessment process as easy as possible to understand and perform. When we roll out the assessment process to courses taught by faculty who have not been working on the Committee, we want them to benefit from our experience by eliminating as much of the uncertainty and frustration with the process as possible
VII. B. General Expectations Related to the MAcc Assessments in Process
The Department of Accounting AOL Committee believes that a single set of measurements is generally insufficient data for requiring changes in the content or teaching of our courses. Results may vary (even from “does not meet expectations” to “meets or exceeds expectations”) across sections of the same course even though the content and approach are the same, and even though the instructor is the same. This is especially true when the assessment process has just been implemented. Thus, we do not expect that the results of our first attempt at assessments for the MAcc Program will lead us to make significant changes in the Program. We expect it will take several years for us to accumulate sufficient valid and reliable data that would support mandating such changes. However, we also expect that our experiences and the results will help us make improvements in the AOL process fairly quickly.
VII. C. Future Plans
As a result of our experience with the AOL process in the College so far, the Coles College AOL Council believes that improved communication to the students of the Learning Goals and
Objectives and how they relate to their learning experience at the Coles College is desirable. The “puzzle pieces” of the Goals are already displayed in our building. However, we would like to help the students make a better connection between the Goals and their courses. Beginning Fall 2006, all Coles College faculty will be expected to include the Goal “puzzle pieces” in their syllabi, and to indicate which Goals are being emphasized in the course. An example of this is in Exhibit VII-1. (All Goals are addressed in ACCT 8990 because of its mission as the Capstone course. Other courses would only focus on some of the Goals.)
Also, beginning Fall 2006, the Coles College curriculum committees will require proposals for new courses to include a description of the relevant AOL Learning Goals and Objectives, so that faculty will explicitly think about how the courses fit into our Programs and serve student learning.