Web-based quantitative analysis and reporting of program outcome coverage and student performance. Paul Van Halen

(1)

Web-based quantitative

analysis and reporting of

program outcome coverage

and student performance

_{Paul Van Halen}

(2)

OUTLINE

• Background • Motivation • Implementation – Course Assessment – Program Assessment • Results • Challenges • Future Development • Conclusions

(3)

BACKGROUND

• Portland, OR

– located in the Willamette valley

– where the Willamette river joins the Columbia – 70 miles from the Pacific coast

– on the Oregon/Washington state line

– with Mount Hood, the highest peak in Oregon as its backdrop

• With 2 million people, the greater Portland metro area accounts for over half the 3.6 million

population of Oregon

• Contrary to popular belief it never rains in Portland!

(4)

(5)

BACKGROUND

• Oregon University System

– 81,000 students

– 7 universities: PSU, UO, OSU, OIT and 3 regional schools

– OSU and PSU offer engineering degrees

• Portland State University

– Located in downtown Portland

– With 24,000 students, the largest enrollment in the state system

(6)

BACKGROUND

• Maseeh College of Engineering and Computer Science

– Electrical & Computer Engineering – Mechanical & Materials Engineering – Civil & Environmental Engineering

– Engineering & Technology Management – Computer Science

– 70 full time faculty

(7)

BACKGROUND

• Electrical & Computer Engineering

– 2 ABET accredited programs: Electrical Engineering and Computer Engineering – 18 faculty

– 290 bachelor students

– 164 MS and MEng Students – 42 PhD students

(8)

MOTIVATION

• ABET 2000 criteria • False starts

– College initiatives

– University assessment initiative

• What are our program objectives? • What are our program outcomes?

• Once we have program outcomes how do we find out how well we address them and how do we perform?

(9)

MOTIVATION

• How does this connect with the grades we give our students?

• We should be able to combine the grading data to assess our stated program

outcomes both in coverage and student performance

• Course outcomes and course assessment are a prerequisite

• How do we get from course assessment to program assessment?

(10)

PROGRAM ASSESSMENT

• Assessment in most cases is still focused on course assessment

• Convincing faculty to participate in this endeavor is a challenge

• Good course assessment does not necessarily compile to a meaningful

program assessment, unless the course assessment was planned with program assessment in mind.

(11)

IMPLEMENTATION

• Program objectives • Program outcomes

• Program outcomes to objectives mapping • Course outcomes

• Course outcomes to program outcomes mapping

(12)

PROGRAM OBJECTIVES

The electrical engineering program has the following educational objectives:

• Knowledge. To provide our students with a

broad knowledge base in the fundamentals and techniques of the engineering sciences, required for engineering careers in a changing technical environment, to prepare them for successful participation in multi-disciplinary teams.

• Application. To provide our students with an in-depth knowledge of the concepts, techniques and tools of the electrical engineering discipline and to impart the ability to apply their proficiency to engineering design and problem solving.

(13)

PROGRAM OBJECTIVES

• Innovation. To provide our students with the ability and desire to continually renew their

education in a rapidly developing discipline,

enabling them to participate in the research and development of the discipline and to realize their full potential throughout their career.

• Community. To ensure awareness of:

a) the need for personal development, both in discipline related aspects and in terms of understanding the

impact of the profession on social and environmental issues.

b) the importance and benefits of personal involvement in professional societies and local communities.

(14)

PROGRAM OUTCOMES

• Added more specific EE and CpE outcomes to the existing (a) – (k) ABET outcomes.

• Created a mapping from the program outcomes to the program objectives.

(15)

(16)

NUMERICAL COURSE TO PROGRAM

OUTCOME MAPPING

• Instructor in charge of a course has to provide a list of course outcomes as part of a standard syllabus for the course

• Instructor also has to provide a weighted mapping of course to program outcomes

– Provides relative importance of each course outcome

– Provides information on how a particular course addresses program outcomes

– Mapping is normalized with respect to credit hours

(17)

(18)

(19)

TECHNICAL DETAILS

• Use Apache Software Foundation tools as a framework:

– Tomcat is a servlet container/server

– Cocoon is a web development framework which provides us with a component-based ability to dynamically generate, transform, serialize web pages from a variety of data sources through “component pipelines”

– Forrest is a Cocoon-based publishing

framework that transforms data from various sources into a unified presentation, including web site navigation/decoration

(20)

TECHNICAL DETAILS

• Data entry and manipulation:

– Altova Authentic

– Authentic is free but the development tools are not

– Authentic is a Windows only program – A web-based version is IE only

• Data is stored as xml files on a web server • Course data (syllabus, outcomes,

assessment) are stored in the web server directories of individual faculty

(21)

(22)

COURSE GRADING AND ASSESSMENT

• Excel spreadsheet

• For every item that is part of the grade

– Maximum grade

– Mapping to course outcome – Grade for every student

(23)

COURSE GRADING SHEET

distribution A 85 A- 80 B+ 75 B 70 B- 62 C+ 55 C 50 C- 45 D+ 40 D rubrics _T1Q 1 T1Q 2 T1Q 3 T1Q 4 T1Q 5 T2Q 1 to 2 T2Q 3 to 5 T3Q 1 T3Q 2 T3Q 3 T3Q 4 T3Q 5 T4Q 1 T4Q 2 T4Q 3 T4Q 4 T4Q 5 PA PB max_grade 3 3 3 3 3 6 9 3 3 3 3 3 3 3 3 3 3 12 12 outcome 1 1 1 1 1 2 2 3 4 3 4 4 4 6 6 5 5 5 6 S1 3 0 3 3 0 1.2 6.6 3 3 0 0 3 0 0 3 0 3 9.2 8 S2 0 0 0 3 0 6 6 0 3 2.4 3 3 0 2.4 3 0 3 10 9.4 S3 3 3 0 3 0 6 6 3 1.2 3 0 0 0 3 3 3 3 10 9.4 S4 0 3 0 3 0 3 6 3 3 3 3 3 0 0 3 0 3 6.4 7 S5 3 3 0 0 0 2.4 1.2 3 3 3 3 3 3 0 3 0 3 6.4 7 S6 3 3 3 3 3 5.4 4.8 0 3 3 3 0 0 0 3 3 3 7.6 7.8

(24)

COURSE OUTCOME ASSESSMENT

C o ur se O u tc o m e 1 C o ur se O u tc o m e 2 C o ur se O u tc o m e 3 C o ur se O u tc o m e 4 C o ur se O u tc o m e 5 … student 1 A B … student 2 B A student 3 A A student 4 F D student 5 C C student 6 D F student 7 B B student 8 B C

(25)

(26)

PROGRAM OUTCOME COVERAGE

• Table compares the coverage of program outcomes according to course syllabi

(required and electives) and assessment data

• EE required courses: 60 credit hours • EE elective courses: 132 credit hours • Is coverage appropriate ?

(27)

EE PROGRAM OUTCOME COMPARISON

Program Outcome (EE) Importance

(required) Importance (elective) Importance (syllab %) Importance (assess %)

(a) An ability to apply knowledge of mathematics, science and _engineering 27.6 70.1 52.3% 57.9% (b) An ability to design & conduct experiments, as well as analyze

and interpret data 1.55 4.35 3.2% 3.1% (c) An ability to design a system, component, or process to meet _{a range of informal to formal descriptions/specifications} 14 21.3 18.9% 15.5% (d) An ability to function on multidisciplinary teams 2.2 1.1 1.8% 1.8% (e) An ability to identify, formulate, and solve engineering _problems 1.75 11.4 7.0% 4.8% (f) An understanding of professional and ethical responsibility 0.65 1.7 1.3% 2.8% (g) An ability to communicate effectively 3.6 5.7 5.0% 3.6% (h) Have the broad education necessary to understand the impact

of engineering solutions in a global and societal context 0.25 0.8 0.6% 1.3% (i) Recognition of the need for & ability to engage in life-long

learning 1.3 2.3 1.9% 2.4% (j) Knowledge of contemporary issues 0 0.2 0.1% 0.0% (k) Ability to use the techniques, skills, modern engineering tools 4.85 10.2 8.1% 4.6%

(28)

PROGRAM ASSESSMENT RESULTS

• Summary of performance for each of the (a) – (k) ABET outcomes

• All course in the EE program for which

data was available (127 credit hours, 3166 student credit hours:

Σ (# of credit hours) x (# of students) • For each of the (a) - (k) outcomes:

– Weight in the overall assessment – GPA

(29)

PROGRAM ASSESSMENT (a)

(a) An ability to apply the knowledge of mathematics, science, and engineering

Importance Percentage of total assessment 57.89 %

GPA 2.47

(30)

PROGRAM ASSESSMENT (b)

(b) An ability to design and conduct experiments, as well as analyze and interpret data

GPA 2.44

(31)

PROGRAM ASSESSMENT (c)

(c) An ability to design a system, component, or process to meet a range of informal to

formal descriptions/specifications

GPA 2.32

(32)

PROGRAM ASSESSMENT (d)

(d) An ability to function on multi-disciplinary teams

GPA 3.17

(33)

PROGRAM ASSESSMENT (e)

(e) An ability to identify, formulate, and solve engineering problems

GPA 2.13

(34)

PROGRAM ASSESSMENT (f)

(f) An understanding of professional and ethical responsibility

GPA 2.03

(35)

PROGRAM ASSESSMENT (g)

(g) An ability to communicate effectively

GPA 2.96

(36)

PROGRAM ASSESSMENT (h)

(h) Have the broad education necessary to understand the impact of engineering

solutions in a global and societal context

GPA 1.67

(37)

PROGRAM ASSESSMENT (i)

(i) Recognition of the need for, and an ability to engage in life-long learning

GPA 2.48

(38)

PROGRAM ASSESSMENT (j)

(j) Knowledge of contemporary issues

Importance Percentage of total assessment 0 %

GPA NaN

(39)

PROGRAM ASSESSMENT (k)

(k) An ability to use the techniques, skills, and modern engineering tools necessary for

engineering practice

GPA 2.69

(40)

CHALLENGES

• Faculty participation

• Version control OR How useful is feedback information when we change the feedback system?

(41)

FUTURE

• Convert data storage from file based in

faculty web directories (~/public_html) to a postgresql database

• Replace Altova Authentic with Cocoon

forms for data entry/update to make the whole process truly web based

(42)

CONCLUSIONS: Closing the loop …

• The quantitative program assessment outlined in this session provides fact based, data driven feedback in the

continuous program improvement cycle

mandated by ABET, and facilitates a better quality management decision process