Great Plains POD Conference,

(1)

Academic Service

Academic Service--Learning in a

Learning in a

Research Methods Course: Using

gg

Student Work to Assess Course Goals

Renee Michael Renee Michael Department of Psychology Department of Psychology R kh r t Uni r it R kh r t Uni r it Rockhurst University Rockhurst University Kansas City, Missouri Kansas City, Missouri

(2)

What can be learned from this

sample of student reflection?

(3)

Introduction

Research on Outcomes

A picture has emerged that service

A picture has emerged that service--learning has

A picture has emerged that service

A picture has emerged that service learning has

learning has

positive effects on attitudes, values, and beliefs

(see for example, Eyler & Giles, 1997, Giles &

(

p

y

(

p

y

Eyler, 1994; Markus, Howard & King, 1993;

Waterman, 1997).

Student self

Student self--reports (Eyler and Giles, 1999)

reports (Eyler and Giles, 1999)

showed decreased stereotyping and increased

l

f

l h

k d i h l

i h

l

f

l h

k d i h l

i h

tolerance of people they worked with along with

an increased appreciation of other cultures.

(4)

Research on Outcomes

As the increased effect on student attitudes has been As the increased effect on student attitudes has been

demonstrated others have begun to

demonstrated others have begun to call for increased call for increased attention to effects on academic components

attention to effects on academic components. .

A small but growing body of evidence has foundA small but growing body of evidence has found

A small but growing body of evidence has found A small but growing body of evidence has found

positive effects but has measured student learning positive effects but has measured student learning mostly through student and instructor self

mostly through student and instructor self--reportsreports

(Eyler & Giles 1997; Hammond 1994; Markus (Eyler & Giles 1997; Hammond 1994; Markus (Eyler & Giles, 1997; Hammond, 1994; Markus, (Eyler & Giles, 1997; Hammond, 1994; Markus, Howard & King, 1993).

Howard & King, 1993).

While informative, student and instructor perceptions While informative, student and instructor perceptions

may or may not be linked to actual learning.

may or may not be linked to actual learning. Few Few studies have

studies have

directly

examined the effects of examined the effects of service

(5)

Research on Outcomes

One way to measure student learning is through One way to measure student learning is through final course yy gg gg final course

grades

grades. But, the findings from those studies have been . But, the findings from those studies have been somewhat inconclusive (see review in Eyler, 2000).

somewhat inconclusive (see review in Eyler, 2000).

That is That is perhaps not surprisingperhaps not surprising, considering that course grades , considering that course grades

are a global measure comprised of several possible behaviors are a global measure comprised of several possible behaviors (e g effort motivation tutoring etc ) and may not be sensitive (e g effort motivation tutoring etc ) and may not be sensitive (e.g., effort, motivation, tutoring, etc.) and may not be sensitive (e.g., effort, motivation, tutoring, etc.) and may not be sensitive to the service

to the service--learning experience alone. learning experience alone.

It may be more fruitful to It may be more fruitful to examine specific exam questions examine specific exam questions

and/or student journal responses

and/or student journal responses whose content is tied to the whose content is tied to the academic course objectives of the service

(6)

Why student work?

Advantages to using work from students in

existing class assignments:

Already haveAlready have not an addnot an add on to class assignmentson to class assignments

Already have Already have –– not an addnot an add--on to class assignments.on to class assignments.

Closer examination/insight into student learning in Closer examination/insight into student learning in

your class What are students learning? Where do your class What are students learning? Where do your class. What are students learning? Where do your class. What are students learning? Where do they need more practice?

they need more practice?

Data (student work) is grounded in your disciplineData (student work) is grounded in your discipline ––Data (student work) is grounded in your discipline Data (student work) is grounded in your discipline

you need not become a social science researcher to you need not become a social science researcher to address course outcomes.

(7)

Course & Students

The course is a Research Methods course in the The course is a Research Methods course in the

Psychology Department at Rockhurst University, a Psychology Department at Rockhurst University, a small, comprehensive university in Kansas City,

small, comprehensive university in Kansas City, Missouri.

Missouri.

The course is required for Psychology majors and The course is required for Psychology majors and

i ll f d i h S h P h l

minors, as well as for students in the Speech Pathology minors, as well as for students in the Speech Pathology program. It is often recommended for Business majors program. It is often recommended for Business majors as well.

as well.

(8)

Community agency & project

The primary project for this course involves the construction, The primary project for this course involves the construction, i l t ti l i d ti f lt f

i l t ti l i d ti f lt f titi ii

implementation, analysis, and reporting of results from a

implementation, analysis, and reporting of results from a questionnaire questionnaire study.

study.

The serviceThe service--learning element of the course is interwoven into the semester as learning element of the course is interwoven into the semester as gg the primary project on which students work. During the teaching of this the primary project on which students work. During the teaching of this course,

course, students worked at a community centerstudents worked at a community center that provided a variety of that provided a variety of services to persons in a low

services to persons in a low--income, innerincome, inner--city neighborhood. The center city neighborhood. The center director wanted to assess the services the center already provided as well as director wanted to assess the services the center already provided as well as y py p learn what additional services were desired by clients.

learn what additional services were desired by clients.

Students were involved in all aspects of carrying out this researchStudents were involved in all aspects of carrying out this research from from th n tr ti n f th q ti nn ir nd it dmini tr ti n t th d t

th n tr ti n f th q ti nn ir nd it dmini tr ti n t th d t the construction of the questionnaire and its administration to the data the construction of the questionnaire and its administration to the data analyses and reporting the findings to the community center.

(9)

Course Goals Assessed

The primary learning goals are for students to learn the The primary learning goals are for students to learn the

basic concepts of empirical research methodology, to basic concepts of empirical research methodology, to engage in the research process and to understand the engage in the research process and to understand the engage in the research process, and to understand the engage in the research process, and to understand the strengths and limitations of different research

strengths and limitations of different research approaches.

approaches.

Th l l d h i i

Th l l d h i i k i l dk i l d

The goals related to their communityThe goals related to their community--work include work include

experiencing interactions with those in need and experiencing interactions with those in need and deepening their understanding of the relationship deepening their understanding of the relationship

between poverty and society. A related course goal was between poverty and society. A related course goal was to assist students in seeing the real world benefits of

to assist students in seeing the real world benefits of carefully constructed research.

(10)

Academic Learning Goals

(11)

Academic Outcomes:

Assessed from Survey

Survey Results show

Survey Results show::

Survey Results show

Survey Results show::

The community participation aspect of this

course helped me to see how the subject

matter I learned can be used in everyday life.

(4 4)

(4.4)

The community work I did helped me to

b

d

d h l

d

di

i

b

d

d h l

d

di

i

better understand the lectures and readings in

this course. (4.0)

(12)

Academic Outcomes:

Assessed from Student Exams

Application component of exam

(exam points) (exam points) 1919--20 2120 21--2222 2323--2424 2525 ( p ) ( p )

* Service

* Service--learning

learning

course

4 1 4 5

* Non service

* Non service--learning

learning

course

3 8 3 0

* Grade assigned:

(13)

Academic Outcomes: Academic Outcomes:

Assessed from student responses to reflection Assessed from student responses to reflectionpp

question question

Question

Question: What ethical issues should be considered while: What ethical issues should be considered while Question

Question: What ethical issues should be considered while : What ethical issues should be considered while conducting this project?

conducting this project?

Concept Concept

Concept

Concept Concept

Concept

Concept Concept Concept

listed elaborated w/example

N

f

N

f

No. of concepts

11

00 2 students

2 students

00

22 2 students 3 students

2 students 3 students

00 33--44

5 students 1 student

1 student

33 44

5 students 1 student 1 student

(14)

Academic Outcomes: Academic Outcomes:

Assessed from student responses to reflection Assessed from student responses to reflection pp

question question

Question: Now that the project for Seton Center is mostly complete, what Question: Now that the project for Seton Center is mostly complete, what advice would you give to the Center if they were to do this project again. advice would you give to the Center if they were to do this project again. Speak to

Speak to any 4 pointsany 4 points concerning the survey construction, administration, or concerning the survey construction, administration, or analysis that might improve the information they receive.

analysis that might improve the information they receive.

Explicit connections made to course concepts with Explicit connections made to course concepts with

examples provided:

examples provided: 2 responses2 responses

Good suggestions provided, but connections not Good suggestions provided, but connections not

explicit:

explicit: 9 responses9 responses

Listing of concepts only, no examples or elaboration:Listing of concepts only, no examples or elaboration:

2 responses 2 responses

(15)

Summary of Academic Learning

Goals

Information from Survey

Information from Survey: students reported the

: students reported the

Information from Survey

Information from Survey: students reported the

: students reported the

community project helped them to learn better

Information from Exams

Information from Exams: students did do

: students did do

Information from Exams

Information from Exams: students did do

: students did do

better on application portion of exam compared

to students in a non S

to students in a non S L section of the course

L section of the course

to students in a non S

to students in a non S--L section of the course.

L section of the course.

Information from Reflective writing

Information from Reflective writing: ability to

: ability to

i

i l /S

i

i l /S L

L

j

integrate course material w/S

integrate course material w/S--L project not

L project not

clearly demonstrated.

(16)

Personal and Career Skills Goals

(17)

Personal and Career Goals:

Assessed from Survey

Survey data show

Survey data show::

Survey data show

Survey data show::

Doing work in the community helped me to Doing work in the community helped me to

become aware of my personal strengths and become aware of my personal strengths and become aware of my personal strengths and become aware of my personal strengths and weaknesses. (3.8)

weaknesses. (3.8)

The community work in this course assisted me in The community work in this course assisted me in cc y wy w s cs c s ss s ds ss s d

clarifying my career plans. (3.0) clarifying my career plans. (3.0)

(18)

Personal and Career Goals:

Assessed from student responses to reflection

d

p

question

Question: Question: Integration of information across different situations QQ Integration of information across different situations gg

is one of the goals of a college education. Describe any is one of the goals of a college education. Describe any information, skills, and self

information, skills, and self--knowledge you have gained through knowledge you have gained through hi j

hi j this project this project..

Skills Gained

Skills Gained::

Practical or jobPractical or job--related (7 responses)related (7 responses)

(19)

Summary of Personal & Career Goals

Information from Survey

Information from Survey::

Information from Survey

Information from Survey::

The survey results showed only a mild endorsement The survey results showed only a mild endorsement

of gained personal or career

of gained personal or career--related skillsrelated skills butbut

of gained personal or career

of gained personal or career related skills, related skills, but . . .but . . .

Information from Reflective writing

Information from Reflective writing::

St d tSt d t t lt l l b t pl b t p ll dd

Students wrote clearly about personal and Students wrote clearly about personal and

interpersonal skills enhanced and how the interpersonal skills enhanced and how the

experience could be directly related to their career experience could be directly related to their career experience could be directly related to their career experience could be directly related to their career goals.

(20)

Summary & Conclusions

Again

Again –– what does student work give us that

Again

Again what does student work give us that

what does student work give us that

surveys cannot?

Close examination of student learning in my courseClose examination of student learning in my coursegg yy

Deeper understanding of to what extent a course Deeper understanding of to what extent a course

goal or objective is reached goal or objective is reached

What are advantages of survey?

Relatively quick form of assessment Relatively quick form of assessment

Set up to be easily summarizedSet up to be easily summarized