MPA Program Assessment Report Summer 2015
Introduction: This was the second full year for doing learning outcomes assessment based on the 2009 NASPAA accreditation standards and conducting our exit interviews using an on-line survey. For the first time, the program assessed all five of NASPAA’s core competency areas. In addition, the program tested a new cultural competency component, to be fielded in its final form in the Fall 2015 semester. Finally, one MPA student (as part of a directed study in program evaluation) surveyed her peers on student support services. That data is also included in this report.
Exit Interview Results
Student Assessment of Learning:
Students believe they have learned quite a bit from our program. They were asked to evaluate what they knew relative to our learning outcomes at the beginning of our program and what they believe they could do now. Table 1 presents the contrast.
Table 1: Self-Assessment of Learning
Before Joining the
Program
At the Conclusion of
the Program Properly identify human, organizational, and
resource issues in a case or situation and develop
an appropriate course of action 3.11 4.67
Describe how public policy decisions are made 2.56 4.44 Identify and explain the political, social, and
economic contexts in which public policy decisions
are made 2.44 4.56
Collect and analyze data essential to understanding
and addressing problems 3.11 4.56
Properly identify the legal and/or constitutional issues in a case or situation and develop a course of
action that is legal and constitutional 2.67 4.44
Properly identify a public administrator’s
responsibility to elected officials and the general public in a case or situation and develop a course of
action or solution that considers these issues 2.56 4.67
Effectively communicate in writing 4.11 4.56
Effective communicate orally in a public presentation 3.78 4.56
Work in a team 4.22 4.33
In two open-ended questions, students were asked to identify additional skills or knowledge gained from the program, along with what they wished they had learned.
Additional skills learned included the ability to better spot weaknesses in work, skills in public budgeting, and learning about the political process.
The “wished for” list included the injection of more “real-world” experiences into coursework.
Student Assessment of Advising
Feedback on advising was generally very positive, as shown in Table 2 below:
Table 2: Student Evaluation of Advisor
Strongly disagree
Disagree Neither Agree nor
Disagree
Agree Strongly Agree Willing to meet with me
at a time and place that
was convenient 0 0 1 3 5
Helpful and courteous 0 0 0 4 5
Helped me develop a course schedule that fit
my needs 0 1 0 3 6
Responded to my
questions in an accurate
and timely manner 0 0 0 3 6
Recommending the Program to Others
Eight of 9 students indicated they had recommended our program to people they know.
The one who did not criticized overuse of powerpoint by some professors (unnamed).
Student Assessment of Quality of Instruction
Students were asked to assess the quality of instruction for both core and elective courses. Table 3 and 4 present the results.
Table 3 (next page) presents the results for core courses.
Table 3: Student Assessment of Quality of Instruction, Core Courses
Strongly disagree
Disagree Neither Agree nor Disagree
Agree Strongly Agree PADM 6000, Survey of
Public Administration 0 1 0 2 6
PADM 6050, Constitutional and
Administrative Law 0 0 0 2 7
PADM 6100,
Organizational Theory
and Behavior 1 0 2 6 2
PADM 6200, Human
Resource Management 1 0 2 6 3
PADM 6300, Public
Budgeting 0 1 0 5 3
PADM 6600,
Quantitative Methods 0 0 2 3 4
PADM 6650, Public
Policy Analysis 0 0 1 3 5
PADM 6750, Program
Evaluation 0 1 0 3 5
Table 4 (next page) presents the results for elective courses. The numbers vary since students make different choices as to which electives to take. Overall, however, the feedback was very positive and clearly comparable to the results for core courses.
Table 4: Student Assessment of Quality of Instruction, Elective Courses
Strongly disagree
Disagree Neither Agree nor Disagree
Agree Strongly agree
Did not take
the course
PADM 6020, GIS 0 0 0 2 2 5
PADM 6030, Grant Writing &
Administration 0 0 0 1 1 7
PADM 6250, Introduction to
Urban Planning 0 0 1 0 0 8
PADM 6301, Financial
Management for
Nonprofits 0 0 0 1 3 5
PADM 6302, Nonprofit
Management 0 0 0 1 2 6
PADM 6350, Emergency
Management 0 0 0 0 0 9
PADM 6351, Introduction to Homeland
Security 0 0 0 2 5 1
PADM 6352, The Unconventional
Threat 0 0 0 0 1 8
PADM 6550, Human Services
Administration 0 0 0 0 1 8
PADM 6700, Urban
Government
Administration 0 0 0 0 2 7
Learning Outcomes Assessment
This academic year, we were scheduled to conduct assessments in five courses:
PADM 6100, 6050, 6600, 6650, and 6900. This academic year, all of NASPAA’s 5 core
competencies were evaluated. Table 5 summarizes the results of learning outcomes assessment this academic year.
Table 5: Results of Learning Outcomes Assessment in Courses
Course NASPAA
Competency
Learning Outcome Assessment Method & Standard
Results PADM 6100 To lead and
manage in public governance
Students will be able to compare and contrast a given pair of leadership theories
80% of students will correctly answer exam question
11 of 12 students scored 4 or better on the rubric, meeting the standard.
PADM 6600 To analyze, synthesize, think critically, solve problems and make decisions
Students will be able to analyze data
Students shall score one unit better on the post-test than the pre-test
Four of six students scored 2- 4 units better on the post-test than on the pre-test;
one remained the same and the sixth student’s
performance declined by 3 points. However, the class average was one unit better on the post-test than the pre-test.
PADM 6050 To articulate and apply a public service perspective
Students will be able to properly identify legal and Constitutional issues
80% of students will correctly identify legal and
Constitutional issues
All students scored 16 or more on the case assignment;
all students met the standard.
PADM 6650 To participate in and contribute to the policy making process
Students will be able to describe the policy-making process, including its actors
80% of students will correctly answer exam question
This learning outcome was NOT met; of the 10 papers evaluated only 6 exceeded the standard grade of 70%
PADM 6900 To
communicate and interact with a diverse and changing workforce and citizenry
Students will be able to demonstrate the ability to
effectively
communicate and interact with others in a team project, negotiation, or an internship
80% of students will demonstrate the ability to effectively communicate and interact with others by achieving a score of 30 or higher on peer evaluations or site supervisor evaluations
All students exceeded the standard of 30 on site supervisor evaluations; the class average was 49.14.
This round of learning outcome assessments was both instructive and helpful. As a result of assessment, we learned that we need to focus more attention on the structure and content of PADM 6650. This is the second year in which students in this course did not successfully meet the expected learning outcomes standard. Since two different instructors were involved, we can conclude it was not the instructor. Both instructors did, however, use the same text and general approach to the class. Clearly, this needs significant work.
Cultural Competency Component
In the 2013-2014 academic year, the faculty completed a review of our curriculum to see how we were covering cultural competency (diversity) issues. Carrizales (2010) recommends curriculum components include 1) knowledge of local and national
demographics, societal disparities, and policy and legal issues surrounding diversity; 2) a self-reflection component that includes societal biases; 3) a skills-based component that focuses on communication skills; and 4) a community-based component that includes internships and other experiential learning offerings. Our review found that we were missing the self-reflection component.
As a result, the MPA Director contacted Dr. Lorraine Evans from the Office of Diversity and Inclusion who put together an on-line component that could be used with PADM 6000 as the self-reflection component and would introduce students to cultural
competency. That component was tested with the PADM 6100 class in the Spring 2015 term. The results were overwhelmingly positive.
The pre/post test was adapted from the Clinical Cultural Competency Questionnaire (CCCQ) developed by Robert C. Like, MD, MS, Professor and Director of the Center for Healthy Families and Cultural Diversity, Department of Family Medicine, UMDNJ-Robert Wood Johnson Medical School. The 14-question survey addressed perceived overall self-efficacy as well as knowledge, confidence, skills and self-awareness, all key constructs in cultural competency, as well as overall perspectives on the topic and general demographics. The results are depicted in Table 6 on the next page.
Table 6: Pre/Post Test Score Comparison Pre-test
mean
Post-test mean
Change I consider myself to be culturally
competent
2.89 3.56 0.67*
I feel confident in my ability to
utilize communication skills to improve service encounters with people of
diverse backgrounds (e.g. different ages, genders, ethnicities, language skills, and socioeconomic statuses).
3.11 3.89 0.78**
It is important for public administration professionals to receive training in cultural competency.
3.56 3.78 0.22
*=p<.05, **= p<.01 N = 9
In addition to the pre-test/post-test, students also wrote a self-reflective essay focused on cultural learning, unconscious bias and promoting an inclusive workplace. The goal was to encourage critical reflection on the topic and required students to connect the material to real world problems. It was a pass/fail assignment and all students
successfully completed the essay.
The cultural competency component will be given in conjunction with PADM 6000, Survey of Public Administration, beginning with the Fall 2015 semester and continue as a component in that class.
Student Services
As part of a directed study version of PADM 6750, Program Evaluation, Sophie Spencer developed, conducted, and analyzed a survey of her fellow students to assess student satisfaction with campus services. A summary of broad results is presented in Table 7 below; a more in-depth discussion of bookstore, library, and career services is provided after Table 7 on the next page.
Table 7: Summary of Results from Student Services Survey
Item Finding(s)
Perceptions of campus safety
13 of 15 respondents said they felt “safe” or “very safe” on campus Classroom
quality
11 of 15 respondents were “satisfied” or “very satisfied” with the quality of classrooms
Food service 3 of 14 respondents were “dissatisfied” or “very dissatisfied” with food services; another 6 were neutral
Class registration
All 15 respondents reported registering for classes on-line; 13 of them were “satisfied” or “very satisfied” with that process
Student financial accounts
All 15 respondents know how to access their student financial account on-line; 12 of them were “satisfied” or “very satisfied” with that service.
Admissions process
10 of 15 students were either “satisfied” or “very satisfied” with the admissions process
Financial aid While students had a high degree of familiarity with where to apply, satisfaction rates were lower. Of the 12 students who had used financial aid, only half (6) were either “satisfied” or “very satisfied.”
Advising All respondents knew who their advisor was, and 8 of 15 claimed to communicate “frequently” or “very frequently” with her. 12 of 15 were either “satisfied (4)” or “very satisfied (8)” with the advisor.
Internships All respondents knew who to contact for information about an
internship. The 8 who had had an internship all said they were “very satisfied (6)” or “satisfied (2)” with the experience itself, the help getting the internship, and the advising process for the internship as Bookstore: 14 out of 15 respondents knew where the bookstore was, but half of them claimed they never went there. When asked to rate the bookstore experience, the students were fairly satisfied with service but highly dissatisfied with price. 10 of 15 respondents said they were “dissatisfied (3)” or “very dissatisfied (7)” with the cost of books.
Library: Interestingly, respondents were just as likely to use the library on campus as to use its on-line services, as illustrated in Table 8.
Not at all
Sometimes Frequently Very Frequently
Physically going to the library 2 8 4 1
Online access 3 6 4 2
Online librarian assistance 15 0 0 0
On campus librarian
assistance 8 6 1 0
Satisfaction with library services was VERY high, with higher satisfaction rates reported for on campus experiences. For example, 13 of 15 students were either “very satisfied”
or “satisfied” with on campus services, but just 8 of 15 rated online services that way.
Career Services:
While most students know what Career Services does and where it is located, the overwhelming majority are not using those services. As an example, the most
frequently used service was resume development, and just 5 of 15 students had used that service.
Marketing
In the 2013-2014 academic year, the program faculty adopted a marketing plan with an ambitious goal of reaching an enrollment of 45 by August 1, 2015. Although the action steps listed in the marketing plan were pursued, and in most cases completed, this goal will not be met.
Actions Taken:
Program director and/or faculty attended 3 graduate fairs, and TGS carried MPA promotional materials to an additional 3 graduate fairs. One of those fairs was the one held in Atlanta, for the HBCUs in that area.
Promotional materials were acquired. A new brochure was developed and printed in the summer of 2014, and additional materials were ordered in May of this year to assist with next year’s recruitment.
A special handout was developed and taken to graduate fairs to highlight to relative low cost of our program compared to others.
The website was overhauled to make it more current and attractive
Approval was received (in concept) for two new additions—a nonprofit certificate and a BA to MPA option—that could help attract new students to the program Additional Steps Needed:
Separation of the MPA Director duties from the Center for Public Service will take place July 1 of this year. This will allow the new MPA Director to focus
exclusively on the program—and student recruitment.
The current MPA Director has been moved back into teaching required courses at the undergraduate level. This will help ensure undergraduates in our
department have contact with MPA faculty on a more regular basis—and should help recruitment.
Cultural Competency (Diversity)
The specific status of each of the goals set in our plan is attached to this report. The significant accomplishments this year included:
Raising $15,000 for a new scholarship to assist students in defraying the cost of completing an MPA degree
Improvements in the advising process to include mandatory in-person advising for new students and follow-up on their performance in the first semester of classes
Creation of a self-awareness component (discussed earlier under learning outcomes assessment)
Overall Conclusions
Graduating students believe they learned quite a bit from the program, were quite satisfied with advising, and the overwhelming majority are recommending the program to others.
Once again, students in PADM 6650 did not achieve the desired learning
outcomes. It is recommended that this course receive a complete overhaul (new text and approach) to ensure students learn what we would like them to learn.
Generally, students are satisfied with services on the campus. However, students could benefit from more contact with career services. The faculty should discuss was to integrate career services into courses, perhaps as brief presentations, to help “sell” students on the importance of using those services.
Recruiting of students needs significant attention. Steps taken this year should lay the groundwork for better results next year; however, the new director will need to make this a priority.
Cultural Competency Status
Strategy Tactic(s) Responsible Party Status as of Spring
2015 Target for recruitment
undergraduate institutions and
programs popular with minority groups.
Attend at least two graduate school/career fairs sponsored by institutions or consortia with large populations of minority students.
Ribando/Hammond Complete
Highlight the low cost of our program compared to other institutions
1. Develop a handout comparing our tuition to others for use at recruiting fairs.
2. Put information about our costs prominently on our website.
Ribando Complete
Locate donors to fund scholarships for MPA students
1. Community Advisory Board member(s) will lead effort to raise funds for Ralph H. Walker
scholarship
2. Use awareness event for the Center for Public Service to raise funds for the scholarship (10/24)
Clayton & Ross/Ribando Complete; $15,000 raised. Development of process/policy for scholarship underway.
Expected award for Fall 2015
Make in-person
advising mandatory for students in their first semester
Schedule in-person advising with each new student in the first semester to assess career goals, experience and courses needed to support those goals.
Ribando Complete
Improve
communication
between instructors of first semester courses and the MPA Director
Director to contact instructors to explicitly ask for feedback on the performance of new students in their first semester at least one.
Ribando/Hammond/Miller Complete
Strategy Tactic(s) Responsible Party Status as of Spring 2015
Raise awareness of our new status as part of an R1 institution
Participate as board members, committee members, or active volunteers in national and regional professional associations
All On-going
Actively recruit a diverse part-time faculty
Use community connections to search for part-time faculty from under-
represented groups to teach in the program or serve as guest speakers
Ribando List of guest speakers
provided to Dr.
Hammond for use in 6650; no current need for additional part-time faculty
Add a self-reflection component to the curriculum to promote cultural competency
Adapt the existing on-line component on cultural competency designed for
healthcare professionals to public administration. Incorporate that into the first semester experience for all new students and make completion mandatory for graduation.
Ribando Incorporated into PADM
6100 as pilot; fully operational in Fall 2015
