Program Review Guidelines
Undergraduate Programs
Purpose
Program review is an opportunity for programs without discipline specific accreditation or approval to engage in:
Synthesis and evaluation of the degree to which the program meets intended operational and student learning outcomes;
Strategic planning to guide continuous improvement of the program;
Peer review to gather constructive feedback from the field regarding the quality of the program.
It is coordinated by the Office of Institutional Effectiveness every 7 years for programs without periodic, discipline specific accreditation or approval. New programs undergo program review in their 4th year in preparation for post-implementation review by the Alabama Commission on Higher Education and every 7 years thereafter. All programs that do not have periodic, discipline specific accreditation or approval are required to participate. The process generally takes about 12 months to complete; however, the maximum amount of time allotted to complete the self-study is three semesters (not including summer). Appendix A includes a timeline for completing the self-study. The department’s and program’s annual assessment and data reporting should be sufficient to identify the 3-year trend data needed to complete the self-study. Outcomes and conclusions identified in the self-study and/or the external review should be used to develop subsequent annual assessment plans for programs, departments, and
colleges/schools, as appropriate. The Program Review process includes:
A self-study of the program;
A review of the self-study by a qualified external reviewer;
A review and approval of the self-study by the Dean, AVP for Institutional Effectiveness, and the SVPAA/VPHS, as appropriate;
Contents of the Self-Study:
1) Title sheet indicating the program’s official name and CIP code 2) Signature page (see Appendix B)
3) Self-Study Summary
Summary of significant conclusions drawn from the program review process to include identification of the program’s strengths, areas in need of improvement, and proposed goals and actions that will inform future program development. Summary may be presented in the form of a SWOT analysis to include strengths, weaknesses, opportunities, and threats. Data/evidence, which should be
detailed in the full document, must be used to justify conclusions. 4) Program Overview
Description of the program to include the mission statement, a brief history of the program, its role within the College/School and University, and how it supports the University’s mission and strategic plan.
A degree plan that lists all courses and the semester in which these courses are typically offered.
A curriculum map that reflects connections between student learning outcomes, assessment processes, and required courses.
5) Program Resources (see Appendix C for required data/evidence).
To what extent is student enrollment adequate and/or appropriate to support the program?
To what extent are faculty qualifications and numbers sufficient to support the program?
To what extent is the program utilizing technology to improve course/program delivery and/or enhance instruction?
To what extent are facilities, space, technology, and other resources sufficient to support the program (Data provided, in part, from USA facilities survey)?
To what extent do students and faculty have access to the library resources and services that support the program? Also, please identify what resources and services may be lacking, if any.
Other important resource areas as identified by program faculty
6) Student Learning Outcomes Assessment: Please consider data you have collected for each of the last three years on the program’s student learning outcomes
individually and as a unit (see Appendix D for required data/evidence and additional questions to prompt faculty inquiry).
What can faculty conclude about the quality of student learning?
What can faculty conclude about the ability of their students to progress through and graduate from the program?
What can faculty conclude about the preparedness of their graduates to enter the workforce and/or continue to post-baccalaureate education?
7) Approval from Chair and/or Dean and Sr. VPAA indicating self-study is ready for external review (see Appendix B).
8) External review
Minimum qualifications for the external reviewer
Certified reviewer from a professional association or a distinguished colleague in the discipline with regard to teaching and scholarship.
Must not have a conflict of interest (e.g., current/former USA faculty, family member or business partner with any current departmental faculty, etc.). Process for choosing an external reviewer
Prior to identifying potential reviewers, the Dean and Department chair should meet to determine whether the external reviewer will conduct a paper review only or also conduct a site visit. The decision to host a site visit is made by the Dean. At this meeting, the Dean and Department Chair also should clarify expectations for qualifications of the reviewer, expectations for the self-study, and expectations for the site-visit (if applicable).
Department chair and/or faculty identify three potential reviewers and submit curriculum vitas for the three individuals to the Dean.
The Dean makes the selection and notifies the department chair. External reviewer report (See Appendix F for rubric)
While there is no standard format for the external reviewer’s report, he or she should use the rubric to evaluate the self-study and use the 1st and 2nd level headers as an outline to structure the report. Additional areas of review and evaluation (beyond those covered by the rubric) are encouraged and
welcome as appropriate.
9) External reviewer report from the Council on Undergraduate Research (optional and strongly encouraged). Visit
http://www.cur.org/projects_and_services/program_review_services/ for more information or contact USA’s Undergraduate Research Program at
[email protected]. 10) Required Appendices
Chair and program coordinator/director curriculum vitae Course syllabi
Student/Program handbooks, if applicable Required tables (see Appendices C and D)
Other data, evidence, or other supporting documentation as applicable or needed to justify conclusions about programs effectiveness.
Appendix A
Month/Date Activity
July Offices of Institutional Effectiveness (IE) and Institutional Research (IR) prepare program review materials and program specific data.
August (last half)
Program coordinator and/or chair meets with (IE) to review the program review process and schedule.
Chair and/or program coordinator meets with faculty and staff.
Chair and/or program coordinator submits list of potential external reviewers to the Dean for approval.
September
(first half) Dean submits list of approved reviewers back to chair and/or program coordinator. September
(last half) Chair and/or program coordinator secures external reviewer. October –
March Program staff and faculty prepare self-study.
April Program staff and faculty submit the self-study for an internal review and to the Dean, AVPIE, and Sr. VPAA for preliminary approval.
May Chair and/or program coordinator sends the report to the external reviewer.
No Campus Visit With Campus Visit
August (last half)
External reviewer’s report submitted to Dean; Review external reviewer’s comments with the dean, chair/program coordinator, and faculty.
Prepare for campus visit.
September (last business day)
Revise assessment plan, update results and develop actions based on findings of the self-study and the external reviewer’s recommendations. Report in TracDat.
External reviewer campus visit completed.
External reviewer’s report submitted to Dean; Review external reviewer’s comments with the dean, chair/program coordinator, and faculty.
October (last business day)
Submit the final self-study to Dean, AVPIE, and Sr. VPAA for approval.
Revise assessment plan, update results and develop actions based on findings of the self-study and the external reviewer’s recommendations. Report in TracDat. Submit the final self-study to Dean, AVPIE, and Sr. VPAA for approval. November
(last business day)
Send a copy of the approved Self-study to OIE
Send a copy of the approved Self-study to OIE
December (last business day)
Upload report into TracDat as supporting documentation
Upload report into TracDat as supporting documentation
APPENDIX B
THE UNIVERSITY OF SOUTH ALABAMA
Academic Program Review for Undergraduate Programs Self-Study Approval Signature Page
Department: Program:
Title Date Printed Name Signature
Department Chair Dean Assoc. VP for Institutional Effectiveness Sr. VP for Academic Affairs VP for Health Sciences
Appendix C
List of Required Data/Evidence for Program Evaluation Program Review of Undergraduate Programs
Table 1: Program level enrollment trends for the last five years for both full-time and part-time students.
Table 2: Program graduation trends for last five years (headcount) (also ACHE needed for post-implementation review).
Table 3: Program new student enrollments for the last five years (headcount only) (also ACHE needed for post-implementation review).
Table 4: List of identified peer programs (University, degree, and program name) Table 5: List of current program faculty, teaching load, courses most often taught,
degrees, recent scholarly/creative activity (last three years), and usage of high-impact practices (see Appendix E).
Table 6: Participation in high impact practices. Table may include high-impact practices and numbers of students participating or identification of courses in which the practice was used (last two years). For courses, please list course name/number and the semester in which it was taught.
Special Note: For programs submitting ACHE post-implementation reviews in the year following program review, two other conditions also must be addressed; however, data for these two conditions is provided by the Office of Institutional Research.
1. IPEDS degree completion
Appendix D
List of Required Data/Evidence for Student Learning Outcomes Program Review of Undergraduate Programs
All required data/evidence is available either 1) through your annual TracDat reports or 2) from the Office of Institutional Research or Office of Institutional Effectiveness. Program faculty should feel free to include other data in addition to the required elements to support their decisions and/or conclusions in the self-study.
Table1: Program level aggregation of student mastery of student learning outcomes for the last three years (also needed for ACHE post-implementation review).
Table 2: Program level aggregation of student retention (1 and 2 year) and completion (4 and 6 year) rates. For the last three cohorts, as applicable.
Table 3: Program level aggregation of student evaluation of teaching for the last three years.
Table 4: Program level aggregation of post- graduation outcomes assessment for the last three years. This may be include one or more of the following (but not limited to) exit surveys, alumni surveys, employment rates, graduate school acceptance rates, etc. (also needed for ACHE post-implementation review).
Appendix D (cont.)
Questions to prompt inquiry about Student Learning Outcomes Assessment
Student Learning Outcomes Assessment: Please consider data you have collected for each of the last three years on the program’s student learning outcomes individually and as a unit.
What can faculty conclude about the quality of student learning?
Questions to prompt inquiry: Is the standard for achievement set at an appropriate level? Is the standard understood and agreed upon by all faculty? Are there student learning outcomes where students perform particularly poorly or particularly well? Are there student learning outcomes where the standard for achievement should be raised or lowered? Where data are available, how do student compare to students in other USA programs or nationally?
What can faculty conclude about the ability of their students to progress through and graduate from the program?
Questions to prompt inquiry: Do students exhibit difficulty progressing
through the major at any point? If no, what are the things that faculty do that support student progression? If yes, at what points and/or why? Do students exhibit difficulty completing the program? If no, what are the things that
faculty do that support student completion? If yes, at what points and/or why? What can faculty conclude about the quality of their teaching?
Questions to prompt inquiry: If there are areas of student learning identified as needing improvement (improving outcomes or raising standards), is modification/improvement of instruction identified as a possible means to improvement? Have there been efforts made to support and/or improve the quality of faculty instruction? Did program faculty implement any high impact practices and what was their effect, if any, on student learning outcomes, faculty or student engagement, etc. (see Appendix E for a non-exhaustive list of high impact practices)?
What can faculty conclude about the preparedness of their graduates to enter the workforce and/or continued post-baccalaureate education?
Questions to prompt inquiry: Consider the program’s student learning outcomes and how they translate to knowledge, skills, and/or abilities that make your students attractive to potential employers and/or graduate school?
Appendix E
List of High Impact Practices for student success/engagement in undergraduate programs (not exhaustive)
Service-Learning
Undergraduate Research (UCUR) Learning Communities First-Year Experience Capstone Experiences Team-Based Learning Intrusive advisement Study Abroad
Appendix F
Program Review (undergraduate programs): Self-Study Evaluation Rubric (under development)
1—Beginning 2—Developing 3—Good 4—Exemplary
Student Learning Outcomes* (SACS 3.3.1.1)
Clarity and Specificity of SLO’s
No outcomes stated. Outcomes present, but with imprecise verbs (e.g., know), vague description of
content/skill/or attitudinal domain, and non-specificity of whom should be assessed (e.g., “students”)
Outcomes generally contain precise verbs (e.g, rich
description of the content/skill/or attitudinal domain, such as those in Bloom’s Taxonomy, and specification of whom should be assessed (e.g., “graduating seniors in the Biology B.A. program”)
All outcomes stated with clarity and specificity including precise verbs, such as those in Bloom’s Taxonomy, rich description of the content/skill/or attitudinal domain, and specification of whom should be assessed (e.g., “graduating seniors in the Biology B.A. program”)
Orientation of SLO’s No outcomes stated in student-centered terms.
Some outcomes stated in student-centered terms.
Most outcomes stated in student-centered terms.
All outcomes stated in student-centered terms (i.e., what a student should know, think, or do).
Curriculum mapping of SLO’s to courses No activities/courses
listed.
Activities/courses listed but link to outcomes is absent.
Most outcomes have classes and/or activities linked to them.
All outcomes have classes and/or activities linked to them.
Relationship between measures and SLO’s Seemingly no
relationship between outcomes and
measures.
At a superficial level, it appears the content assessed by the measures matches the
outcomes, but no explanation is provided.
General detail about how outcomes relate to measures is provided. For example, the faculty wrote items to match the outcomes, or the instrument was selected “because its general description appeared to match our outcomes.”
Detail is provided regarding objective-to-measure match. Specific items on the test are linked to outcomes. The match is affirmed by faculty subject experts (e.g., through backwards translation).
Types of measures
No measures indicated Most outcomes assessed primarily via indirect (e.g., surveys) measures.
Most outcomes assessed primarily via direct measures.
All outcomes assessed using at least one direct measure (e.g., tests, essays).
Appendix F
1—Beginning 2—Developing 3—Good 4—Exemplary
Student Learning Outcomes* (SACS 3.3.1.1)
Specification of desired results No a priori desired
results for outcomes
Statement of desired result (e.g., student growth,
comparison to previous year’s data, comparison to faculty standards, performance vs. a criterion), but no specificity (e.g., students will grow;
students will perform better than last year)
Desired result specified. (e.g., our students will gain ½ standard deviation from junior to senior year; our students will score above a faculty-determined standard). “Gathering baseline data” is acceptable for this rating.
Desired result specified and justified (e.g., Last year the typical student scored 20 points on measure x. The current cohort underwent more extensive coursework in the area, so we hope that the average student scores 22 points or better.)
Data collection and research design integrity No information is
provided about data collection process or data not collected.
Limited information is provided about data collection such as who and how many took the assessment, but not enough to judge the veracity of the
process (e.g., thirty-five seniors took the test).
Enough information is provided to understand the data collection process, such as a description of the sample, testing protocol, testing conditions, and student motivation. Nevertheless,
several methodological flaws are evident such as unrepresentative sampling, inappropriate testing conditions, one rater for ratings, or mismatch with specification of desired results.
The data collection process is clearly explained and is appropriate to the specification of desired results (e.g., representative sampling, adequate motivation, two or more trained raters for performance assessment, pre-post design to measure gain, cutoff
defended for performance vs. a criterion)
Additional validity evidence (as appropriate to the measure being used) No additional
psychometric properties provided.
Reliability estimates (e.g., internal consistency, test-retest, inter-rater) provided for most scores, although reliability tends to be poor (<.60). Or, author states how efforts have been made to improve reliability (e.g.,
Reliability estimates provided for most scores, most scores are marginal or better (>.60).
Reliability estimates provided, most scores are marginal or better (>.60). Plus, other evidence given such as relationship of scores to other variables and how such relationship strengthens or weakens argument for validity of test scores.
Appendix F
Student Learning Outcomes* (SACS 3.3.1.1)
Presentation of results
No results presented Results are present, but it is unclear how they relate to the outcomes or the desired results for the outcomes.
Results are present, and they directly relate to the outcomes and the desired results for outcomes but presentation is sloppy or difficult to follow. Statistical analysis may or may not be present.
Results are present, and they directly relate to outcomes and the desired results for outcomes, are clearly presented, and were derived by appropriate statistical analyses.
History of results
No results presented Only current year’s results provided
Past iteration(s) of results (e.g., last year’s) provided for some assessments in addition to current year’s.
Past iteration(s) of results (e.g., last year’s) provided for majority of assessments in addition to current year’s.
Interpretation of results No interpretation
attempted
Interpretation attempted, but the interpretation does not refer back to the outcomes or desired results of outcomes. Or, the interpretations are clearly not supported by the methodology and/or results.
Interpretations of results seem to be reasonable inferences given the outcomes, desired results of outcomes, and methodology.
Interpretations of results seem to be reasonable given the outcomes, desired results of outcomes, and methodology. Plus multiple faculty interpreted results (not just one person). And, interpretation includes how classes/activities might have affected results.
Dissemination of results No evidence of
communication
Information provided to limited number of faculty or
communication process unclear.
Information provided to all faculty, mode and details of communication clear.
Information provided to all faculty, mode and details of communication clear. In addition, information shared with others such as advisory
committees, other stakeholders, or to conference attendees.
Appendix F
Student Learning Outcomes* (SACS 3.3.1.1)
Use of results for improvement/modification (Student learning and development) No mention of any
improvements.
Examples of improvements documented but the link between them and the assessment findings is not clear.
Examples of improvements (or plans to improve) documented and directly related to findings of assessment. However, the improvements lack specificity.
Examples of improvements (or plans to improve) documented and directly related to findings of assessment. These improvements are very specific (e.g., approximate dates of
implementation and where in curriculum they will occur.) Use of results for improvement (assessment process)
No mention of how this iteration of assessment is improved from past administrations.
Some critical evaluation of past and current assessment,
including acknowledgement of flaws, but no evidence of improving upon past
assessment or making plans to improve assessment in future iterations.
Critical evaluation of past and current assessment, including acknowledgement of flaws; Plus evidence of some moderate revision, or general plans for improvement of assessment process.
Critical evaluation of past and current assessment, including
acknowledgement of flaws; both present improvements and intended improvements are provided; for both, specific details are given. Either present improvements or intended improvements must encompass a major revision.
*Nearly all of the rubric for Student Learning Outcomes is borrowed from James Madison University’s Assessment Progress Template (APT) Evaluation Rubric © 2013 by Keston H. Fulcher, Donna L. Sundre, & Javarro A. Russell.
Appendix F
1—Beginning 2—Developing 3—Good 4--Exemplary
Program Resources
Use of technology for instruction (3.4.12)
TBD TBD TBD TBD
Use of high-impact practices (3.3.1.1) No students or faculty are
actively engaged in high-impact practices at least once annually.
Very few students or very few faculty are engaged in high-impact practices at least once annually.
Approximately half of the students and faculty are engaged in high-impact
practices at least once annually.
Nearly all of the students and faculty are engaged in high-impact practices at least once annually.
Faculty qualifications (3.7.1)
TBD TBD TBD TBD
Faculty resources (3.5.4) At least 25% of course hours in the major are taught by faculty holding an appropriate terminal degree.
At least 50% of course hours in the major are taught by faculty holding an appropriate terminal degree.
At least 75% of course hours in the major are taught by faculty holding an appropriate terminal degree.
All or nearly all course hours in the major are taught by faculty holding an appropriate terminal degree.
Adequacy of facilities (space) (3.11.3)
TBD TBD TBD TBD
Adequacy of facilities (technology) (3.4.12)
TBD
TBD TBD TBD
1—Beginning 2—Developing 3—Good 4--Exemplary
Overall Evaluation (3.3.1.1)
Major findings and/or conclusions
TBD TBD TBD TBD
Use of review process to drive planning
