Educational Evaluation

A.5. The quality of student learning at ML is monitored through a systematic network of internal and external tools. Over the past six years, as part of the MDiv review, we have been carefully examining the extent to which students have met the goals of the MDiv degree program. We have learned two things: 1.) these goals are largely being met, and 2.) we are confident that we can increase the efficacy of our educational model in a way that is beneficial for students and the communities that they will serve. In light of these findings, we have refined our educational goals, our educational model, and our assessment procedures in order to provide an educational

experience that empowers students to reflect and integrate theoretical learning with the realities of professional ministry in a multiracial, multicultural, and theologically diverse world.

73 In the past, the educational goals of the MDiv degree were measured through a combination of direct and indirect evidence. For Residential students, the core educational goals were measured as follows:

 Religious heritage: GPA, Internship evaluations (self and supervisor)

 Cultural context of ministry: Internship evaluations (self and supervisor)

 Personal and spiritual formation: CPE evaluation, Career Assessment, Internship evaluations

 (self and supervisor)

 Capacity for ministerial and public leadership: CPE; Internship evaluations (self and supervisor)

For students in the modified residency program, the review process entailed examining the following materials:

 Religious heritage: GPA, Internship evaluations (self and supervisor), Integrated paper (written during internship year)

 Cultural context of ministry: Field supervisor evaluations, Monthly reports, Internship evaluations (self and supervisor)

 Personal and spiritual formation: CPE evaluation, Career Assessment, Internship evaluations

 (self and supervisor), Praxis evaluations, Annual review with academic advisor and field advisor

 Capacity for ministerial and public leadership: CPE, Internship evaluations (self and supervisor), Praxis evaluations, integrated paper and evaluation, Annual review with academic advisor and field advisor

As a result of our ongoing analysis of this data, we were aware of both strengths and weaknesses of the existing MDiv program. Residential students reported dissatisfaction on two counts, the cost of the degree and the cumbersome division of academic work and internship experience.

Residential students took academic courses for two years, left for a one year stand- alone

internship, then returned for a year of course work. While we offered integrative seminars for the internship, students found the process to be both expensive and often disruptive. After the year of internship, they were ready for ministry, and often found it difficult to return to academic work.

Students in the modified residency program had greater satisfaction with the cost and logistics of the degree program. They had significantly less student debt, completed their internship and accompanying integrated paper the last year of their degree program, and were ready to move into professional ministry.

In our evaluation of the goals of the MDiv program and the degree to which we met those goals, we also carefully studied the 2008 Alban Study, “Becoming a Pastor: Reflections on the Transition into Ministry” by James P. Wind and David J. Wood and the 2007study of theological education by the Auburn Institute, “How Are We Doing? The effectiveness of theological schools as

measured by the vocations and views of graduates” by Barbara G. Wheeler, Sharon L. Miller, and Daniel O. Aleshire." We noted in the Auburn study the success of traditional models in providing the academic context of and foundations for ministry, but less success in providing education in the skills required for ministerial practice. We noted from the Alban study the need to provide ongoing theological reflection and collegial support as students move to the praxis of ministry.

74 With the help of educational consultant Wayne Goodwin, we enhanced our MPR model to meet both needs, embedding ministerial education in the practice of ministry, and teaching students how to create and work with learning communities, a practice that they can take forward into their professional lives.

In 2011 we received a three year grant from the Luce Foundation. A central component of that grant was the assessment of the efficacy of our current MDiv educational model, as well as

beginning the ongoing assessment of the new model. We worked with FHI 360, an internationally respected team of assessment specialists who assisted with the ongoing design and implementation of Meadville Lombard’s evaluation strategy. With the help of FHI 360 we redesigned the

Meadville Lombard Graduating Student Survey and have conducted a survey of alumni/ae from the past ten years. All graduates in the first cohort will be interviewed by external educational experts (FHI 360) in order to meet two goals: 1.) gather more information about the efficacy of the TouchPoint program as well as identify areas that need improvement, and 2.) gain data that will enable us to revise and improve the survey instrument itself. In the future, alumni/ae will be queried at five-year intervals through surveys, interviews, and focus groups. The evaluation strategy is designed to unpack student experiences in community and congregational settings and the degree to which Meadville Lombard prepared them for the realities they encounter working in religious communities. In order to achieve impartiality with the data, external evaluation will continue to be conducted by FHI 360.

In order to gain baseline data, we conducted a survey of all graduates from the past 10 years. We found that we had provided a strong academic foundation for ministry, but were lacking in developing some of the key dispositions and skills necessary for the practice of ministry in a multicultural, multiracial, and theologically diverse world. The survey had an excellent return rate of 50%, and we found that the quantitative data confirmed the insights derived from our qualitative data on the following counts: MRP students incurred significantly less debt; there was general satisfaction regarding academic preparation; and we were not as effective in shaping dispositions for cross-cultural, collaborative ministerial and public leadership.

Residential MRP

Would choose the same seminary again 62% 83%

Incurred no debt while in seminary 21% 79%

Program very effective/effective in providing comprehensive understanding of religious heritage

48%/46% 55.6%/40.7%

How well did the program prepare you for your current work?

Very effective/very well in some areas but not others

21.2%/67.3% 41.4%/31.4%

Program very effective in cultivating the capacity for ministerial and public leadership

31.4% 48.3%

Program effective in teaching them the practice of collaborative leadership

17.3% 24.1%

Program effective in teaching the practice of cross-cultural leadership

9.6% 17.2%

Program effective in shaping the disposition of working creatively with ambiguity

37% 61%

Program effective in cultivating the ability to thrive in difficult situations

38.5% 46.4%

75 With this baseline data, we were able to begin a rigorous comparison of the efficacy of the

TouchPoint model. Initial data are promising. In May 2012 we surveyed the first graduates of the new program, and are able to contrast the results of their education with those in the MRP

program. The results of this survey are quite promising. They demonstrate a clear improvement in the TouchPoint model of the fundamental goals of integrating the theory and practice of ministry and developing the skills and dispositions for ministerial leadership in a multicultural, multiracial, theologically diverse world.

1. When asked to describe their overall experience at ML, 63.6% of the TouchPoint students said that the educational experience was academically rigorous; 90.9%

said that the internship had been a helpful experience and 72.5% said that they were able

to integrate the theory and practice of ministry. 83.3% of the MRP students found the educational experience to be academically rigorous, 66.7% found the internship to be helpful, yet only 33.3% said that they have been able to integrate the theory and practice of ministry.

2. We asked students how effective their seminary education was in shaping the dispositions that we have identified as critical to collaborative and cross-cultural leadership. The results are telling. This is the degree to which students in each program found the program to be very effective in shaping the following dispositions:

“Very Effective” in Shaping Dispositions

MRP TouchPoint

Theological Imagination, very effective 16.7% 54.5%

Embracing contradiction, very effective 0% 45.5%

Working creatively with ambiguity, very effective

16.7% 45.5%

Thriving in challenging situation 16.7% 54.5%

Addressing conflict directly and constructively

0% 36.4%

Reflecting while acting 16.7% 54.5%

There is also a significant difference between the two programs in building the skills that are constitutive of exemplary ministerial leadership. While we have been far more successful in building these skills in the TouchPoint model, we are not as successful in shaping skills as in shaping dispositions. This will be an area that will be investigated in the focused interviews by our educational consultants. They will ask TouchPoint students what made the fostering of dispositions effective, and what could have been more helpful in building leadership skills.

76

Practicing collaborative leadership 0 16.7% 36.4% 63.6%

Practicing cross-cultural leadership 0 33.3% 9.1% 81.8%

Designing service learning

opportunities for others 0 16.7% 18.2% 36.4%

Facilitating social justice activities 16.7% 16.7% 18.2% 72.7%

Facilitating multifaith awareness and

partnerships 33.2% 0 27.3% 54.5%

Developing a learning community in

ministry setting 25% 0 36.4% 54.5%

In addition to these measures of educational efficacy, we found three areas of concern. 1.) Students in the TouchPoint program stated that Meadville Lombard was only somewhat successful in providing the resources and support needed in finding a community site, in finding a Teaching Pastor and congregation, and providing guidance in moving through the UUA credentialing process. With the help of a newly hired Director of Contextual Ministry, we are working to rectify these shortcomings and hope to see an improvement in the academic year 2012/2013. 2.) Student debt levels are excessive in the TouchPoint program. 54.6% of TouchPoint students have incurred an educational debt of over $50,000 while in seminary. Of the MRP students, 100% have a debt level less than $30,000, and 50% incurred no debt at all. 3.) Only 54.5% of TouchPoint students and 33.3% of MRP students stated that they would choose ML again. We are not sure if this is an accurate measure of student satisfaction or an artifact of how the question was asked. It was not a stand-alone question, but embedded in a choice of multiple ways of characterizing their overall experience at ML. In the focused interviews, the external evaluators will ask this question directly in order to get accurate data.

While the results of the first comparative survey of the TouchPoint model students are promising, we will continue to explore them in four ways:

1. Our educational consultants will conduct formal focused interviews with all graduates of the TouchPoint program in the fall of 2012. This will enable us to explore areas of concern as well as refine the survey for future years.

2. We will conduct surveys and focused interviews with alumni/ae at five year intervals to assess the efficacy of the model as students enter the full time practice of ministry.

3. We will continue the annual survey of graduating students.

4. We will continue the rigorous and ongoing examination of qualitative data.

We have ongoing examination of qualitative data in two forms – a longitudinal study of student outcomes, and an in-process examination of student outcomes. In order to continue longitudinal studies of student outcomes, it is necessary to consistently archive learning artifacts from every student over time. To meet this goal, ML subscribes to LiveText, an electronic forum designed to capture student data. As such, all assignments, faculty feedback, and formational documents (CPE evaluations, Career assessment, Teaching Pastor and site supervisor evaluations) are permanently housed and securely encrypted in this web-based archive. Data is permanently available to

students after graduation should they wish to engage reflectively on their work during the seminary

77 years, and is permanently available for comparative faculty evaluation of learning outcomes.

In addition, there is an ongoing process of weekly, monthly, and annual evaluations of student learning. The faculty uses a variety of narrative and quantitative strategies to ensure the quality of learning in regular courses and in all-school learning convocations. Over the past three years we have revised the evaluative surveys given for the courses, the Fall and January Learning

Convocations, and the extracurricular January intensive learning events. Faculty and staff meet collectively to evaluate learning and service outcomes of classes and convocations with an eye toward improvement and learning integration (DPS.8). Student learning in the Signature Courses is monitored at the micro-level during weekly faculty meetings of Signature Course faculty wherein student performance is considered. The collective Signature Course faculty meets once per month, and twice per semester, to conduct meta-assessments on learning in Signature Courses, ensure curricular coherence, and plan future course interventions in order to meet desired

outcomes. In addition, Teaching Pastors (fieldwork supervisors) evaluate student learning at placement sites twice per year. At the close of each academic year, faculty from each Signature Course review the totality of student learning artifacts – including evaluations from site

supervisors, Congregational Lay Committees, and core courses – and produce individual narratives for each student that evidence areas of accomplishment and areas for future attention. Signature Course faculty then make recommendations about which students are ready to move to the next stage of learning (DPS.9).