A SYSTEMATIC APPROACH FOR DEFINING AND ASSESSING PROGRAM EDUCATIONAL OBJECTIVES AND OUTCOMES

A SYSTEMATIC APPROACH FOR DEFINING AND ASSESSING PROGRAM EDUCATIONAL OBJECTIVES AND OUTCOMES

Nikos J. Mourtos

1 Professor, Mechanical & Aerospace Engineering, San Jose State University, One Washington Square, San Jose, California 95192-0087 [email protected]

Abstract  The USA Accreditation Board for Engineering and Technology adopted recently a new set of criteria for evaluating engineering programs. The paper describes the design and implementation of a systematic process for defining and assessing Program Educational Objectives (criterion 2) and Program Outcomes (criterion 3). The process begins with definition of detailed course learning objectives to address specific Program Outcomes. New, innovative assignments are introduced in several courses to prepare students in the skills specified by these learning objectives. Student performance is assessed in regards to specific outcomes and changes are implemented to increase student achievement in each outcome. The result was significant curriculum improvements. Program Educational Objectives are defined and evaluated using interviews with graduating seniors, alumni and employer surveys, and input from advisory boards.

Index Terms  accreditation, course design, program assessment, program educational objectives, program outcomes.

INTRODUCTION

The USA Accreditation Board for Engineering and Technology (ABET) adopted recently two new criteria for evaluating engineering programs: Criterion 2, Program Educational Objectives (PEOs,) and Criterion 3, Program Outcomes (POs) [1]. PEOs are defined with input from all program constituents and describe the expected accomplishments of graduates during the first several years following graduation. POs, on the other hand, describe what students are expected to know or be able to do by the time of graduation from the program. A systematic process must be in place to assess the achievement of both the POs – before students graduate – and the PEOs – after graduates leave the program. This process needs to be ongoing to ensure the continuous improvement of each program.

The paper describes the design and implementation of such a systematic process in the Aerospace (AE) and Mechanical (ME) Engineering Programs at San Jose State University. The relationship between PEOs, POs, and Course Learning Objectives (CLOs) is illustrated in Figure 1. Following the process of engineering design, one may

view PEOs as a set of mission requirements (specification).

Program Educational

Objectives Program Outcomes Course Learning Objectives

FIGURE 1

RELATIONSHIP BETWEEN CLOS,POS, AND PEOS

Alumni will be able to meet the expecations set for them in the PEOs, if students have the skills described in the POs at the time they graduate. These skills are acquired mostly through the curriculum of each program.

Hence, learning objectives in each course must represent a subset of the skills described in the POs.

PROGRAM EDUCATIONAL OBJECTIVES

PEOs reflect the career and professional accomplishments of graduates during the first several years after graduation.

The process of definition and assessment of the PEOs is illustrated in Figure 2. Input from students, faculty, alumni, employers, and the two advisory boards (one for each program) is used to validate the definition of our PEOs, as well as to assess their achievement. PEOs are revisited periodically to ensure that they continue to reflect current industrial trends.

Both programs are designed to fulfill the University [2], College [3], and Department [4] mission. They provide students with a broad understanding of basic concepts, as well as the contemporary skills required by industry. The coursework includes extensive laboratory experiences and many opportunities for students to work on hands-on, design projects.

The foundation courses provide a basis for professional competence and the required knowledge to focus on a particular specialization upon graduation², in the work environment or in graduate school.

Define

Program Educational Objectives

Employer Input Faculty

Input

Student Input

Advisory Board

Input Alumni

Input

Assess

Program Educational Objectives

PEOs met?

Program is satisfactory

Yes

Design and implement curriculum changes

No

FIGURE 2

DEFINITIONANDASSESSMENTPROCESSFORPEOS.

The PEOs reflect our constituents’ expectations that our graduates should have:

1. A strong foundation in mathematics, basic science and engineering fundamentals, to successfully compete for entry-level positions or pursue graduate studies in AE / ME or related fields.

2. Contemporary professional and lifelong learning skills including hands-on laboratory experience, familiarity with computers, modern software, and information technology, to successfully compete in the local, national and global engineering market.

3. Strong communication and interpersonal skills, broad knowledge, and an understanding of multicultural and global perspectives to work effectively in multidisciplinary teams, both as team members and as leaders.

4. An understanding of the ethical choices inherent in the engineering profession to deal with issues such as public safety, honest product marketing, and respect for intellectual property.

PROGRAM OUTCOMES

ABET Criterion 3 requires engineering programs seeking accreditation to demonstrate that their graduates have:

a. an ability to apply knowledge of mathematics, science, and engineering

2 Three options are offered in each Program: Aerodynamics and Propulsion, Dynamics and Control, Structures and Materials for AE, Mechanical Design, Mechatronics, and Thermal / Fluids for ME.

b. an ability to design and conduct experiments, as well as to analyze and interpret data

c. an ability to design a system, component, or process to meet desired needs

d. an ability to function on multi-disciplinary teams e. an ability to identify, formulate, and solve engineering

problems

f. an understanding of professional and ethical responsibility

g. an ability to communicate effectively

h. the broad education necessary to understand the impact of engineering solutions in a global and societal context

i. a recognition of the need for, and an ability to engage in life-long learning

j. a knowledge of contemporary issues

k. an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice.

Figure 3 shows the process for assessing outcomes. A course coordinator must show evidence that his / her course includes the necessary elements to satisfy a particular outcome and collect / analyze data to show that performance targets are met. Moreover, for each outcome there is a designated outcome champion. Champions validate the evidence presented by course coordinators for individual courses and have the final word on whether the performance of a program is satisfactory with regards to their outcome. They meet with course coordinators and instructors, discuss their findings and make recommendations for course improvements. Outcome champions provide an additional level of accountability and ensure consistency in the process.

Because outcomes are rather comprehensive and difficult to assess as stated, Felder and Brent [5, p.19]

suggest that each outcome be analyzed into elements – different abilities specified in the outcome – and that a set of attributes be defined for each element – actions that explicitly demonstrate mastery of the abilities specified.

These attributes have been defined at one of the 6 levels of Bloom’s taxonomy in the cognitive domain [6] or 5 levels in the affective domain [7]. Two outcome indicators are used to assess student attainment of program outcomes: (a) course performance ratings based on graded student work and (b) student surveys. To satisfy Criterion 3, performance targets were defined as follows:

(a) The scores earned by all students in the assignments and test questions, which pertain to a particular outcome, in each course where this outcome is measured, must be at least 60%³.

(b) The ratings pertaining to this outcome, given by at least 70% of the students in each class surveyed, must be “I agree” on a 3-point Likert scale. If these targets are met in the courses chosen for assessment of an outcome, the

3 Corresponds to a grade of C-, the lowest passing grade in core courses.

outcome is achieved and no further action is needed in this course. When performance targets are met, courses are assessed on a 3-year cycle. When performance targets are not met in a course, improvements are implemented and the course is assessed on a yearly basis until the targets are met.

COURSE DESIGN AND ASSESSMENT

The importance of course design becomes obvious from Figure 1, as CLOs form the foundation for the POs and the PEOs. Our first step was to define detailed and measurable CLOs for all courses, that describe what students should be able to do upon completion of the course. Typically, a course has 30 – 45 learning objectives. Although content specific, each CLO⁴ addresses a specific PO, as shown in Table 1.

TABLE 1

EXAMPLES OF CLOS FROM AE162(AERODYNAMICS). THE RIGHT-HAND COLUMN SHOWS THE POS ADDRESSED BY EACH CLO[8].

Course Learning Objectives PO

27. Design and perform⁵ an experiment to study the performance of an airfoil, analyze and interpret the results from this experiment, compare with analytical / computational predictions and other published experimental data⁶, and explain any discrepancies⁷.

3b 3d 3g 3i 3k 36. Use the method of images to discuss and calculate aerodynamic interference for (a) wings flying in the vicinity of each other (i.e., wing/tail/canard combinations, biplanes, formation flying, etc.), (b) wind-tunnel boundaries, and (c) ground effects.

3a 3e

44. List several examples of regional, national, and / or global contemporary problems related to aerodynamics (ex.

environmental issues, natural resources and energy conservation, etc.) articulate a problem / position statement for each, and explain what makes these issues particularly relevant to the present time.

3d 3g 3h 3i 3j

The next step is to design lectures, assignments, and other course activities that prepare students in the skills described by the CLOs. Some of the new, innovative assignments, which were introduced in several courses for this purpose, are shown in Table 2. Lastly, Figure 3 shows the process of course assessment.

CONCLUSION

The paper described the design and implementation of a systematic process for defining and assessing Program Educational Objectives and Outcomes. The AE and ME Programs at San Jose State University were evaluated in 2005. ABET evaluators found the approach described here

4 Only a few, selected CLOs are shown in Table 1.

5 Outcome 3d is met by the requirement that students work in teams of 3- 4 to design and perform their experiment, as well as to write their report.

6 Outcome 3i is met as students research the literature for published data and other resources.

7 Outcome 3g is met as students submit a full laboratory report for each experiment.

most comprehensive and expressed their satisfaction that it can indeed be used to ensure the continuous improvement of the programs.

TABLE 2

NEW COURSE ASSIGNMENTS DESIGNED SPECIFICALLY TO SATISFY CRITICAL AREAS OF THE POS.

Course assignment Courses in which assignment was introduced

PO Students design the

experiments they will perform in the various laboratories [9].

ME113-Thermodynamics ME114-Heat Transfer ME120-Experimental Methods AE162-Aerodynamics AE164-Compressible Flow

3b

Students discuss economic, environmental, social, political, ethical, safety, liability, and manufacturability constraints in their design of aircraft / spacecraft.

AE170A&B-Aircraft / Spacecreft Design

3c

Students are taught team skills and required to assess formally the performance of their teammates using specific criteria.

ME120-Experimental Methods AE162-Aerodynamics AE164-Compressible Flow AE170A&B-Aircraft / Spacecreft Design ME195A&B-Senior Design Project

3d

Students identify, formulate, and solve open-ended problems. Some of these problems involve integration of material from two or more courses [10].

ME111–Fluid Mechanics ME113-Thermodynamics ME114-Heat Transfer AE162-Aerodynamics AE164-Compressible Flow AE165–Flight Mechanics AE167-Aerospace Propulsion

3a 3e

Students research, present, and discuss in class safety, ethics, and liability issues in AE.

AE170A&B-Aircraft / Spacecreft Design

3f 3h

Students research, present, and discuss in class contemporary engineering applications and their impact in a global and societal context [11].

ME111–Fluid Mechanics ME113-Thermodynamics ME114-Heat Transfer AE162-Aerodynamics AE164-Compressible Flow AE165–Flight Mechanics AE167-Aerospace Propulsion

3h 3j

R

[1] Criteria for Accrediting Engineering Programs, Effective for Evaluations During the 2005-2006 Accreditation Cycle, Engineering Accreditation Commission, Accreditation Board for Engineering and Technology, <http://www.abet.org/forms.shtml>.

[2] San Jose State University mission statement,

<http://www.sjsu.edu/about_sjsu/mission/>.

[3] SJSU College of Engineering mission statement,

<http://www.engr.sjsu.edu/overview.php>.

[4] SJSU Department of Mechanical & Aerospace Engineering mission statement, <http://www.engr.sjsu.edu/mae/mae%20mission.php>.

[5] Felder, R.M., Brent, R., “Designing and Teaching Courses to Satisfy the ABET Engineering Criteria”, ASEE Journal of Engineering Education, vol.92, no.1, January 2003, pp.7-25.

[6] Bloom, B.S., Taxonomy of Educational Objectives, Handbook 1, Cognitive Domain. New York: Addison Wesley, (1984).

[7] Bloom, B.S., Karthwohl, D.R., Massia, B.B., Taxonomy of Educational Objectives, Handbook 2, Affective Domain. New York:

Addison Wesley, (1984).

[8] An example of course learning objectives / program outcomes matrix from the SJSU AE162 – Aerodynamics course,

<http://www.engr.sjsu.edu/nikos/courses/ae162/AE162LO.htm>.

[9] Du, W.Y., Furman, B.J., Mourtos, N.J., On the ability to design engineering experiments, lead paper, proceedings, 8^th UICEE Annual Conference on Engineering Education, February 2005.

[10] Mourtos, N.J., DeJong-Okamoto, N., Rhee, J., Open-Ended Problem-Solving Skills in Thermal-Fluids Engineering, Invited Paper, Global Journal of Engineering Education, vol.8, no.2, 2004.

[11] DeJong-Okamoto, N., Rhee, J., Mourtos, N.J., Incorporating the Impact of Engineering Solutions on Society into Technical Engineering Courses, Invited Paper, Global Journal of Engineering Education, vol.9, no. 1, 2005.

ABET + PROGRAM FACULTY:

Define Program Outcomes.

OUTCOME CHAMPIONS:

Break down each outcome into elements.

Define outcome attributes for each element.

PROGRAM FACULTY:

Define outcome indicators and performance targets.

Performance targets met?

COURSE COORDINATORS:

Generate student survey including questions for each

of the outcomes addressed in their

course.

OUTCOME CHAMPIONS:

Write student survey questions for each outcome based on attributes

Outcome satisfied.

Yes No

PROGRAM FACULTY:

Identify courses that address this outcome.

PROGRAM FACULTY:

Select courses to be assessed for this outcome.

COURSE COORDINATORS:

Collect / organize course material from each of the selected courses (syllabus, student work,

assignment / test scores for each student).

COURSE COORDINATORS: Analyze data

COURSE COORDINATORS:

Administer student surveys.

OUTCOME CHAMPIONS +

COURSE COORDINATORS:

Recommend / implement curriculum improvements

as needed.

FIGURE 3

OUTCOME ASSESSMENT FLOWCHART

Define CLOs.

List outcomes addressed in the course.

List course activities / assignments / tests that address each outcome.

At the end of the course, when student work has been graded, add for each student the points for all assignments / tests that pertain to

a particular outcome (some assignments may address more than one outcome). Repeat the

process for all outcomes.

Define performance targets (ex. 70% of the students perform at the 70% level in each

outcome addressed in the course).

Performance target met for each outcome?

Create student surveys with questions from all the outcomes addressed in the course.

Define targets for survey responses (ex. 70%

of respondents "agree" in each question).

Target met for each question of the survey?

Recommend course improvements in content / delivery

as needed.

Implement course improvements in the next course

offering.

CLOs and associated POs are met; course is satisfactory.

Build higher student confidence.

Yes Yes

No

No Course Design

FIGURE 4

COURSEASSESSMENTFLOWCHART