Assessment and Evaluation of Student Learning and Curricular Effectiveness

University of Illinois at Chicago / College of Pharmacy (M/C 874)

15. Assessment and Evaluation of Student Learning and Curricular Effectiveness

University of Illinois at Chicago / College of Pharmacy (M/C 874)

15. Assessment and Evaluation of Student Learning and Curricular Effectiveness

As a component of its evaluation plan, the college or school must develop and carry out assessment activities to collect information about the attainment of desired student learning outcomes. The assessment activities must employ a variety of valid and reliable measures systematically and

sequentially throughout the professional degree program. The college or school must use the analysis of assessment measures to improve student learning and the achievement of the professional

competencies.

The college or school must systematically and sequentially evaluate its curricular structure, content, organization, and outcomes. The college or school must use the analysis of outcome measures for continuous improvement of the curriculum and its delivery.

2. College or School's Self-Assessment

The college or school develops and carries out assessment activities to collect information about the attainment of desired student learning outcomes. The

assessment activities employ a variety of valid and reliable measures systematically and sequentially throughout the professional degree program.

Satisfactory

The college or school's evaluation of student learning determines student

achievement at defined levels of the professional competencies, in aggregate and at the individual student level

Satisfactory

The college or school uses the analysis of assessment measures to improve student learning and the achievement of the professional competencies.

Satisfactory

The college or school systematically and sequentially evaluates its curricular structure, content, organization, pedagogy, and outcomes.

Satisfactory

The college or school uses the analysis of outcome measures for continuous improvement of the curriculum and its delivery.

Satisfactory

The college or school has developed a system to evaluate curricular effectiveness. Satisfactory The college or school ensures the credibility of the degrees it awards and the

integrity of student work.

Satisfactory

The college or school has mechanisms to assess and correct underlying causes of ineffective learning experiences.

Satisfactory

The college or school's assessments include measurement of perceived stress in faculty, staff, and students, and evaluate the potential for a negative impact on programmatic outcomes and morale.

Satisfactory

3. College or School's Comments on the Standard Focused Questions

A description of formative and summative assessments and measures used to evaluate teaching and learning methods and curricular effectiveness, including nontraditional pathway(s) leading to the Doctor of Pharmacy degree (if applicable)

A description of the assessment measures and methods used to evaluate student learning and, achievement at defined levels of the professional competencies and educational outcomes, both in aggregate and at the individual student level

How achievement of required competencies by all students is assessed and assured on completion of the program

Comparisons with national data and selected peer-group programs (include a description of the basis for the peer-group selection) and trends over time

How feedback from the assessments is used to improve student learning, outcomes, and curricular effectiveness

The mechamisms in place to assess and correct causes of ineffective learning experiences, including the measurement of perceived stress in faculty, staff, and students and evaluation of the potential for a negative impact on programmatic outcomes and morale

How the college or school is applying the guidelines for this standard in order to comply with the intent and expectation of the standard

Any other notable achievements, innovations or quality improvements

Interpretation of the data from the applicable AACP standardized survey questions, especially notable differences from national or peer group norms

(School comments begin here)

Assessment and evaluation of student performance and curricular effectiveness are critical to curricular quality improvement efforts. An assessment map (Appx 3.2) and plan (Appx 3.4) to evaluate student achievement, teaching and learning methods, and curricular effectiveness have been developed.

Formative and summative evaluations are utilized to evaluate each area. Specific assessment plans are developed for the RPHARM program (Appx 3.5) and strategic plan (Appx 3.6). An admissions assessment plan is currently in development.

For individual students, achievement is assessed via student self-assessment and embedded

assessments within core coursework for continuous curriculum improvement. Students perform several formative self-assessments during IPPE (Appx 14.1). Throughout APPEs, students maintain a portfolio which provides them additional opportunities for formative and summative self-assessments (Appx 14.2).

Until 2012, students had performed self-assessments of their mastery of UIC Professional Competencies

& Outcome Expectations using an internally developed online tool (App 15.1). Due to technical issues, that tool is no longer available online; paper versions are now used. We are exploring using E*Value for self-assessment (Standard 12). Individual student achievement is evaluated through assessments within individual courses (e.g., quizzes, exams, other graded assignments), final course grades, IPPE/APPE evaluations, and maintenance of a satisfactory GPA. To maintain the integrity of these assessments, guidelines were developed for administering exams, academic integrity was defined, and potential consequences of academic dishonesty were described (Appx 15.2-15.3). Because the curriculum is mapped to terminal competencies, students who pass all required courses, including IPPE and APPE activities, and maintain a cumulative GPA greater than 2.00 are deemed to have achieved the required competencies and qualify as credible candidates for graduation.

Student achievement on an aggregate level, evaluation of teaching and learning, and curricular effectiveness are performed following each semester and summarized in an annual report from

15. Assessment and Evaluation of Student Learning and Curricular Effectiveness Page 77

the Assessment Committee (AC) (Appx 3.11). Each semester, student performance on exams,

quizzes, and other embedded assessments are evaluated by class and campus in an effort to identify/

correct inequalities. Since expanding to Rockford, 24.8% of embedded assessments significantly differed between campuses, however the differences are small. In most cases, the Chicago campus outperformed Rockford. We continue to evaluate and compare performance across campuses.

NAPLEX and MPJE passing rates are used as a gauge for student achievement (Appx 3.23-3.25). With the exception of 2012, when the passing rate for graduates was 92.5%, the 5-year NAPLEX passing rates for graduates were within (+/-) 0.7% of the national average passing rate (range 94.41–96.48%), exceeding the national average 2009–2011. Since 2010, graduates have performed within (+/-) 0.3%

of the national average for NAPLEX Competency Areas 1, 2, & 3. The passing rates for the MPJE have been equivalent to (2009) or exceeded (2010–2013) the national average passing rate in recent years.

Compared to other Illinois and research-intensive schools, 5-year passing rates on both NAPLEX and MPJE for students compared favorably. NAPLEX quizzes and an optional NAPLEX and law review are now offered to 4th year students. We conclude student achievement is high and comparable to national averages and peer programs, reinforcing the effectiveness and credibility of our curriculum. Currently, NAPLEX data is not available for the first class that graduated from the two campus program in May 2014.

We solicit feedback from several groups, utilizing various mechanisms for individuals to provide programmatic input. AACP survey data obtains input from faculty, graduating students, alumni, and preceptors (Appx 1.3, 9.3, 9.4, 12.4). The majority of faculty, alumni, and preceptors surveyed in 2013 agree we solicit programmatic feedback, are aware of mechanisms to provide feedback, and believe this feedback and other assessment data are used to improve the curriculum. The overwhelming majority of preceptors (89%) agreed or strongly agreed assessment tools provided were suitable for measuring student performance. All students surveyed agreed or strongly agreed the PharmD program fosters in them the importance of reflection and personal plans for improvement. Students were also strongly supportive of the PharmD curriculum. Students, faculty, and staff are also surveyed on their experiences to assess stressors and to obtain additional feedback. All staff completing the Staff Experience Survey assessed their stress level in the previous month as low/manageable. The majority felt confident in their ability to handle stress, cope with responsibilities, and control irritation.

Similarly, the Experience Survey of the P1 and P2 classes in 2012 (Appx 15.4) revealed the majority of students reported feeling “stressed” fairly often/very often, but characterized their typical stress level as low/manageable. The survey does suggest the stress level increases from P1 to P2 year with less P2 students feeling confident in their ability to handle stress compared to P1 students and more P2 students believing the stress level negatively impacts their performance and morale. However, we recognize this may be class specific and not related to coursework in P1 or P2 years. In 2014, a Pharmacy Environment Questionnaire and the Perceived Stress Scale (PSS) were administered to faculty, staff and students. The results showed faculty/staff stress levels are in the low range (mean score 18.3/56);

the top two most stressing factors for students were examinations/grades and competition for grades for all classes (Appx 15.5-15.6). Measurements of perceived stress amongst faculty, staff and students are performed annually to evaluate and correct ineffective learning experiences. Discussions regarding stress management occurred at the 2014 annual faculty retreat.

Assessment of teaching and learning methods and curricular effectiveness is obtained through various evaluations at semester’s end. Course coordinators complete an End-of-Course Report (Appx 3.9) where feedback regarding student performance and curricular improvement is provided. Similarly, at the conclusion of each didactic course, IPPE and APPE, students complete evaluations of the course

coordinators (Appx 3.7), instructors (Appx 3.8), as well as IPPE/APPE sites and preceptors (Standard 14). Student response rates typically exceed 80%. Feedback from these evaluation tools is used by the instructors, course coordinators, preceptors, the AC and the EPC to make modifications to course content and/or teaching/learning methods.

Student forums (Appx 15.7) are held mid-semester to provide opportunities to give feedback to the Dean and course coordinators regarding curricular issues. The Dean’s Forum is an open discussion for all students conducted by the Dean. Students ask questions related to student experience, including the curriculum. Each class also holds a student forum to provide feedback to the course coordinators each semester. The Associate and Assistant Deans for Academic Affairs and Student Affairs as well as the EPC Chair, Team Leaders, and other members of College leadership participate.

Team Leaders are another conduit of feedback between students, course coordinators, and the EPC.

Team Leaders are assigned to a particular class (e.g., P1, P2, or P3) and serve as resources for course coordinators and students. They attend EPC meetings and provide monthly reports to the committee regarding new or on-going issues related to the curriculum.

Several noteworthy changes to the curriculum were made since the 2007 Self-Study. The most impactful changes include expansion to Rockford, implementation of Echo360 and related distance technology, and ongoing curricular revision efforts. While implementation of distance technology is closely tied to the Rockford expansion, it has also provided faculty with additional teaching and learning methods to employ in the classroom. For example, this technology has been used to “flip” the classroom to provide more active learning opportunities for students during scheduled class time. Additionally, online electives are now offered taking advantage of this technology for course delivery.

The ongoing curricular revision process is the most in-depth curricular change made since 1997. The process started in 2009 through participation in AACP Curricular Change Summit. Over the next 2 years, we obtained input from various stakeholders. Since 2012, we completed 2 curricular revision phases.

In Phase I, a Curricular Revision Steering Committee and 6 Subcommittees consisting of faculty from each department, students, and alumni were formed. Each subcommittee provided recommendations regarding foundational curriculum decisions (Appx 15.8). In Phase II, curricular revision process convened a smaller committee (2 faculty from each department, 1 experiential education faculty and 1 Rockford faculty) to develop a curricular framework, using Phase I recommendations. During Phase II, the committee updated the curricular map, identified gaps and redundancies and addressed these in the final proposal (Appx 15.9). Innovative aspects of the curricular revision proposal include changes to prerequisites, IPPE and APPE, addition of new courses, the concept of modular design for selected courses, flexibility to pursue joint degree programs, and inclusion of competency assessments at various points throughout the curriculum to assess student achievement and curricular effectiveness.

Throughout the process, the EPC has held town hall meetings and presented to each department to engage faculty and solicit feedback. The faculty has approved new competencies, outcomes, and changes to the prerequisites. Phase III of curricular revision is ongoing and consists of developing specific course content and delivery.

4. College or School's Final Self-Evaluation

Compliant

Compliant with Monitoring Partially Compliant Non-Compliant

15. Assessment and Evaluation of Student Learning and Curricular Effectiveness Page 79

5. Recommended Monitoring

(School comments begin here)

University of Illinois at Chicago / College of Pharmacy (M/C 874) 16. Organization of Student Services

The college or school must have an organizational element(s) devoted to student services. The administrative officer responsible for this organizational element must oversee and coordinate the student services of the college or school.

2. College or School's Self-Assessment

The college or school has an organizational element(s) devoted to student services. Satisfactory The organizational element(s) devoted to student services has an administrative

officer responsible for overseeing and coordinating them.

Satisfactory

The budget assigned to student services is sufficient to provide needed services. Satisfactory The college or school has an ordered, accurate, and secure system of student

records which are confidential and maintained in compliance with the Family Educational Rights and Privacy Act (FERPA).

Satisfactory

Student services personnel are knowledgeable regarding FERPA law and its requirements.

Satisfactory

The college or school provides students with financial aid information and guidance, academic advising, career-pathway and other personal counseling, and information about post-graduate education and training opportunities, e.g., residencies,

fellowships, and graduate school.

Satisfactory

The college or school offers access to adequate health and counseling services for students. Appropriate immunization standards exist, along with the means to ensure that such standards are satisfied.

Satisfactory

The college or school has policies in place so that students who have off-campus classes or pharmacy practice experiences fully understand their insurance coverage and where and how to access health and counseling services.

Satisfactory

The college or school has a policy on student services, including admissions and progression, that ensures nondiscrimination as defined by state and federal laws and regulations, such as on the basis of race, religion, gender, lifestyle, sexual orientation, national origin, or disability.

Satisfactory

The college or school ensures that students in all degree program pathways and geographic locations have equal access to and a comparable system of individualized student services (e.g., tutorial support, faculty advising, counseling).

Satisfactory

3. College or School's Comments on the Standard Focused Questions

A description of student services offered and, if applicable, how the college or school ensures that students in all degree program pathways and geographic locations have equal access to and a comparable system of individualized student services (e.g., tutorial support, faculty advising, counseling)

A description of the sections of the student handbook that deal with specific requirements of the standard and guidelines

16. Organization of Student Services Page 81

How the college or school provides students with financial aid information and guidance, academic advising, career-pathway and other personal counseling, and information about post-graduate education and training opportunities

How the college or school is applying the guidelines for this standard in order to comply with the intent and expectation of the standard

Any other notable achievements, innovations or quality improvements

Interpretation of the data from the applicable AACP standardized survey questions, especially notable differences from national or peer group norms

(School comments begin here)

The Office of Student Affairs (OSA), the Office of Diversity and Inclusion (ODI), and the Office of Professional Development (OPD) work collaboratively to provide leadership in the development and provision of student services, especially toward the professionalization and socialization of our pre-pharmacy and professional students. An organizational chart depicting the Provision of Student Services and the corresponding responsible person(s) is found in Appendix 16.1.

Dr. Thomas TenHoeve, Associate Dean, OSA, oversees and leads student services in the COP. In addition, he actively participates in AACP, especially on admissions issues, and has been directly involved with PharmCAS since its inception. His CV can be found in Appendix 16.2.

OSA oversees, coordinates and administers multifaceted activities and services (e.g., recruitment programs, administration of admissions and academic progress, maintenance of records, organization of academic advising and career pathway counseling, verification of completion of degree requirements, linkage with university-wide student services). Dr. TenHoeve, in conjunction with Dr. Kristen Goliak, Director of Experiential Education in the PMPR Department, oversee the annual criminal background checks and drug screenings.

OSA personnel and those dealing with confidential data obtain FERPA certification. The university requires personnel dealing with student records undergo standard online training and pass an associated quiz. All confidential materials are maintained on university secured servers. University policy about FERPA and tutorial can be viewed here: http://tigger.uic.edu/depts/oar/uic_faculty_staff/

student_records_policy.html.

While the COP Admissions Committee (AdC) recommends student acceptance or rejection, OSA provides oversight and administration of the admissions process. OSA advises applicants, processes applications materials (PharmCAS and supplemental), and helps the AdC determine appropriate

applicant screening methods. OSA provides extensive information to prospective students via its website at www.uic.edu/pharmacy/student_affairs and through recruitment visits (e.g., high schools, colleges, universities). Disseminated information includes the PharmD program and curriculum description, COP, program prerequisites, and pre-pharmacy guides (pre-pharmacy course listing) for most schools in Illinois and bordering states. Printed materials are also made available to prospective students.

OSA plans and implements all orientation programs for admitted students. These programs make students aware of campus and COP resources. OSA informs incoming students about professionalism and appropriate behavior, the Honor Code, and general expectations of COP students. The new student orientation also includes the White Coat Ceremony and orientation to information technology resources provided by COP’s Information Technology unit.

Electronically, the OSA provides documents to current students. Examples include, the Student Handbook (available online at the following link: http://www.uic.edu/pharmacy/student_affairs/

current_students/StudentHandbook.pdf), the PharmD curriculum, joint curricula, class schedules, Pharmacy Student Council (PSC) minutes, registration information, scholarship opportunities, and financial aid. OSA provides advising to first-year students and encourages their selection of a faculty advisor/career mentor at the beginning of P1 spring semester. Career advising has recently been expanded under the newly created OPD.

The Scholarship and Awards Committee (SAC) selects scholarship and award recipients. The OSA and the Office of Advancement and Alumni Affairs (OAAA) provide oversight and administration of the scholarship and awards process; coordinating efforts with the SAC by providing an on-line application for scholarships. OAAA is also responsible for the organization of the annual Scholarship and Awards Ceremony and is assisted by OSA. Other than annual COP scholarships and awards, the OSA does not handle student loans or financial aid. These are handled by the Office of Student Financial Aid (OSFA).

Students are referred to OSFA for questions and guidance regarding financial aid information. OSFA has a designated individual for COP students, and s/he is introduced to the incoming students during student orientation.

Any COP student who has a documented disability, as defined by Section 504 of the Rehabilitation Act of 1973 and the Americans with Disabilities Act (ADA) of 1990, is referred to the campus Disability Resource Center (DRC). Students with disabilities requiring accommodations must be registered with the DRC. The DRC writes accommodation letters for students and OSA works with students by having them self-identify to their course coordinators to ensure appropriate accommodations. The COP makes

In document Condensed Self-Study Report of University of Illinois at Chicago (Page 75-89)