defined for his or her degree program and area of concentration.
2.7.a. Description of the procedures used for monitoring and evaluating student progress in achieving the expected competencies, including procedures for identifying competency attainment in practice and culminating experiences.
Beyond the satisfactory completion of required and elective courses, MPH students are assessed using the following methods. Please see ERF 2.6.d for the various modalities used across courses.
Faculty Evaluations of Student Progress:
Classroom Observations: The MPH program combines theory with practice to provide students with immediate applications of new material. Courses, either formally or informally, include laboratory/practice times during the class meeting time. Through the observation of faculty and teaching assistants, on conjunction with their peer classmates, students receive immediate feedback on their understanding and application of important areas of knowledge.
Discussion Boards: Students are often required to reflect on readings and classroom activities through an online discussion board. The discussions occur through the Canvas learning management system.
Examinations: The most traditional form of assessing student learning is through course examinations – weekly quizzes, mid-term exams, and final exams.
Papers: Students are often asked to write papers for various courses to meet specific objectives. Papers can be short one-page mini-cases to longer literature reviews, risk assessments, or policy papers.
Project Work: Small group project work is a common learning experience in the MPH program. Through poster presentations, research projects, and team projects, students are expected to synthesize knowledge and apply skills from several areas to solve a problem and then describe and defend their findings in written and/or oral form.
Culminating Project: The MPH Capstone serves as a means to disseminate and communicate the important work performed throughout the year on a specific public health topic. Thorough oversight by the Capstone course directors, associated faculty, and teaching assistants provides an opportunity for continued student assessment and
feedback on student progress across all three terms. Other Methods of Evaluating Student Progress:
Internship Surveys: As noted in Criterion 2.4, the Internship team, along with the site preceptors, evaluates the students on their in-the-field experience, which is intended to be an application of the competencies gained in the MPH program.
102
Post-graduation Surveys: Surveys sent to students after graduation serve as self- assessment tools measuring the graduate’s performance on the job against program instructional objectives. Surveys are sent to graduates every three years – a time period that avoids recall bias and provides the graduate with adequate work experience to assess their knowledge and skill levels. In 2014, an alumni survey was developed and
distributed to alumni from the classes of 2011-13. Program strengths relevant to core competencies were highlighted by the alumni respondents:
o Critical thinking skills
o Critical appraisal
o Quality improvement
o Data analysis
o Writing skills
o Communication skills
o Understanding of US health care
o Variations
o Patient-centered care
o Working with a variety of people
In addition, alumni from the Class of 2015 were asked to complete a program evaluation one month after graduation. Specific questions were asked about the competencies, and a summary of recent findings is discussed in Criterion 2.7.e.
2.7.b. Identification of outcomes that serve as measures by which the program will evaluate student achievement in each program, and presentation of data assessing the program’s performance against those measures for each of the last three years. Outcome measures must include degree completion and job placement rates for all degrees included in the unit of accreditation (including bachelor’s, master’s and doctoral degrees) for each of the last three years. If degree completion rates in the maximum time period allowed for degree completion are less than the thresholds defined in this criterion’s interpretive language, an explanation must be provided. If job placement (including pursuit of additional education), within 12 months following award of the degree, includes fewer than 80% of graduates at any level who can be located, an explanation must be provided.
Degree Completion
Table 2.7.b.(1) Students in MPH Degree and Degree Completion
Cohort of Students AY11 AY12 AY13 AY14 AY15
2010- 2011
# Students continuing at beginning of this school year
48
# Students withdrew, dropped, etc. 1
# Students graduated 34
Cumulative graduation rate 70.8% 2011-
2012
# Students continuing at beginning of this school year
103
# Students withdrew, dropped, etc. 0 1
# Students graduated 9 23
Cumulative graduation rate 89.6% 60.5% 2012-
2013
# Students continuing at beginning of this school year
4 14 50
# Students withdrew, dropped, etc. 0 0 1
# Students graduated 3 8 40
Cumulative graduation rate 95.8% 81.6% 80.0% 2013-
2014
# Students continuing at beginning of this school year
1 6 9 53
# Students withdrew, dropped, etc. 0 0 0 3
# Students graduated 1 5 6 36
Cumulative graduation rate 97.9% 94.7% 92.0% 67.9% 2014-
2015
# Students continuing at beginning of this school year
0 1 3 14 46
# Students withdrew, dropped, etc. 0 0 0 1 6
# Students graduated 0 0 1 10 29
Cumulative graduation rate 97.9% 94.7% 94.0% 86.7% 63.0%
NOTE: Completion of degree is one year for full-time MPH students and up to 5 years for part-time MPH students, though most part-time students finish within 3 years.
Job Placement
Table 2.7.b.(2). Destination of Graduates by Employment Type - 2013-2015 2013 2014 2015
Employed 38 35 31
Continuing education/training (not employed) 12 13 9
Actively seeking employment 1 2 0
Not seeking employment (not employed and not continuing education/training, by choice)
1 0 0
Unknown 3 0 0
Total 55 50 40
Note: After continued follow-up, the graduates from 2013 and 2014 who were “actively seeking employment” or “unknown” at the time the data were collected (within 12 months of graduation) have since found employment or matriculated for further education. Because it was outside the 12-month window, we left the data in the appropriate cells above as originally reported in our CEPH annual reports for 2013 and 2014.
Table 2.7.b.(3). Outcome Measures for Student Achievement in Competencies and Performance Outcome Measure Target 2012-13 2013-14 2014-15
Retain and graduate high-quality students.
Cumulative graduation rates are above 90%.
97.9% for students entering in AY11 94.7% for students entering in AY12 94% for students entering in AY13 (with 2 remaining students expected to graduate in 2016) Employment rates within 12 months of graduation are above 80%.
104
Offer a unique and competency-based program that addresses the core areas of public health and the challenges faced in the current U.S. healthcare system.
Core courses address five core areas of public health.
2013-14 2014-15 2015-16
100% 100% 100%
Learning objectives and competencies identified in competencies matrix linked to respective course content.
100% 100% 100%
Prepare students for careers in public health practice and research.
Employment rates within 12 months of graduation are above 80%.
93% 96% 100%
85% or higher of alumni report to be adequately prepared or better for current job.
Class of 2011 Class of 2012 Class of 2013
59% 58% 75%
2014 program
evaluation data. N/A N/A See Criterion 1.2.
Conduct relevant capstone projects and present findings to a public health audience [students]. # and variety of capstone projects. 44: (10 intervention; 16 research proposals; 11 policy reviews; 3 program evaluations; 1 quality improvement project; 3 research reports) 33: (5 intervention; 12 research proposals; 6 policy reviews; 3 program evaluations; 2 quality improvement project; 4 research reports; 1 other) Pending # of public capstone presentations. 44 33 pending Advance the population health of communities through course projects, internships, and student-led offerings [students]. # of community-based projects embedded in courses * * 33 # of community organizations * * 39
Plan various activities during Public Health Week each year (began in 2014-15 academic year)
N/A
2.7.c. An explanation of the methods used to collect job placement data and of graduates’ response rates to these data collection efforts. The program must list the number of graduates from each degree program and the number of respondents to the graduate survey or other means of collecting employment data.
105 A pre-graduation survey administered by the Registrar and the Manager of Career Services and Professional Development is sent to all graduating students. To ensure a 100% completion rate, the survey asks for official diploma information in addition to job or further education
information known at that time. For 2015, a 100% completion rate was achieved.
The Manager of Career Services and Professional Developments conducts further follow-up (in person, by phone, by email, or social media) to track placement. She also works closely with the departmental Senior Associate Director of Development and Geisel School of Medicine
Development Office’s development and alumni relations officers, who maintain the official database for MPH alumni through Raiser’s Edge software. The database also contains medical student and housestaff alumni. In 2013, a group of Education staff members worked closely with the Manager of Career Services and Professional Development to update the entire alumni database with employment information and current contact information.
2.7.d. In fields for which there is certification of professional competence and data are available from the certifying agency, data on the performance of the program’s graduates on these national examinations for each of the last three years.
As of June 2015, no graduates have taken the Certified in Public Health (CPH) exam. The MPH’s program leader is asked prior to each CPH testing period to verify graduates of Dartmouth’s MPH program, and none have been listed in the testing database.
2.7.e. Data and analysis regarding the ability of the program’s graduates to perform competencies in an employment setting, including information from periodic assessments of alumni, employers and other relevant stakeholders. Methods for such assessment may include key informant interviews, surveys, focus groups and documented discussions.
Every three years, we conduct an alumni survey. The most recent survey was completed in early 2014 and targeted to the Classes of 2011, 2012, and 2013. A question related to core competency achievement was asked and the results are shown in Table 2.7.b(1).
Table 2.7.b(1): Alumni Evaluation of Student Assessments and Level of Competence How well do you think your instructors’ student assessments (e.g. graded assignments, presentations, papers, quizzes, exams) accurately evaluated your level of competence in each domain?
Underrated my competence Accurately rated my competence Overrated my competence Foundational Knowledge 5.3% (4) 86.8% (66) 7.9% (6) Critical Appraisal 2.6% (2) 93.4% (71) 3.9% (3)
Solving Problems &
Implementing Change 11.8% (9) 81.6% (62) 6.6% (5)
Communication 5.3% (4) 89.5% (68) 5.3% (4)
Professionalism & Leadership 13.2% (10) 82.9% (63) 3.9% (3)
answered question 77 skipped question 12
106 For the 2014 program evaluation, a two-person team (both of whom are employed in the
department and are graduates of our MPH program) collected data over a 90-day period from interviews with alumni, faculty, partners, employers, current students, and prospective students accepted into the MPH program at TDI who chose to enroll elsewhere, and from surveys of recent program graduates. Interviews with employers (including hospitals, consultancies, public health agencies) revealed that they are looking for graduates who can 1) combine many data sources, 2) normalize the data, 3) perform an analysis, and 4) make specific recommendations for action. This requires being facile with multiple research methods and having quantitative skills, and also being able to apply that through a quality improvement (versus research) lens. The employer interviewees also said that our MPH graduates generally have these stills;
however, we heard from public health agencies that our graduates do not always have a working knowledge of public health language, even if they have the skills needed for specific public health roles.
The individuals who were interviewed during the program evaluation also generally felt that TDI is not doing enough to help students succeed after graduation. This was not perceived as much of a problem for mid-career individuals and practicing physicians, but students earlier in their career paths need more support. They felt that more needs to be done to help students identify and pursue the specific skills and competencies to succeed after graduation. They felt that this process should begin on day one of the program and should be guided by a clear and compelling vision answering the question, “We are educating our students to do what?” As a result of this feedback, we have improved our career planning for students and alumni and have incorporated competency development into the career services. This is further discussed in Criterion 4.4. The Phase 1 and Phase 2 Program Evaluation Reports are available in ERF 2.7.e.
2.7.f. Assessment of the extent to which this criterion is met and an analysis of the program’s strengths, weaknesses, and plans relating to this criterion.
This criterion is met.
Strengths:
Students are assessed using a variety of assessment methods (individual and small group projects, poster sessions, exams, written and oral presentations, etc.). These assessments often mimic the employment responsibilities of a public health graduate, and thus, we believe our graduates are well-prepared for the workforce.Graduation rates continue to be high, with an average graduation rate of 93% between 2011-2014 (and it’s too early to include 2015 since many part-time students are in process). Employment data shows that in the past three years, an average of 96.3% of our graduates find employment or begin further education (typically medical school) within 12 months of graduation.
Weaknesses:
We need more formal evaluation mechanisms for tracking our graduates and their competencies in the workplace.
107 While the 2014 program evaluation provided a rich source of data, we also realized we needed a more comprehensive program evaluation plan. As a result and as outlined previously, we have recently contracted with Dartmouth’s Center for Program Design and Evaluation (CPDE) to develop a more formal and systematic approach to track our graduates and assess competencies on an ongoing basis. This approach will include existing and new in-course assessments linked to competencies to assure achievement by students and improved mechanisms to assess student and graduate application of competencies in internship and workplace settings. Specific plans to achieve this include a review of existing processes and measures, revision and development of new measures as needed, and creation of an updated evaluation process and infrastructure to systematically collect, analyze and report data related to competencies, program progress and needed improvements or changes.
109