Examples of Developing a Culture of
Assessment
Developing a Culture of Assessment in the School of Law
The experience of the School of Law, as reported in a paper by the dean and associate dean titled A Funny Thing Happened on the Way to Institutional Assessment (at Pitt Law), captures some common aspects of the evolution of a culture of assessment at Pitt. The paper recounts the process through which the “assessment of student learning outcomes evolved from a University-imposed, administration-centered, uninspiring, anxiety-inducing exercise to a collaborative, creative, mission-clarifying, confidence-building enterprise.” An abridged version of this experience follows.
Pitt Law came to the assessment of student learning outcomes reluctantly, at the prompt- ing of the University administration. Starting conditions were neither ideal nor uncommon: The faculty were resistant to and skeptical of assessment, and those in the dean’s office lacked familiarity with the concept, process, and meth- ods of outcomes assessment and were not much more enthusiastic about the initiative than their colleagues. To meet the University’s requirements while minimizing conflict and stress, the dean took on the responsibility of identifying and defining student learning outcomes (SLOs) as well as designing and implementing methods of assessment. To keep the faculty informed and to obtain faculty input while minimizing the fac- ulty’s time and effort, four faculty “consultants” and the faculty steering committee worked with the dean to develop a plan, with reports to and surveys of the entire faculty at key points.
Faculty were shielded from actually assessing student work, which was done by the dean’s office with the assistance of three faculty librarians.
In several respects, that approach worked. The University’s requirements were met, and elements of the assessment plan were held up as something of a model for other schools. In other respects, however, the initial approach reaped what it sowed. Assessment was treated as a necessary evil, and the faculty, not surprisingly, remained disengaged and unconvinced of the value (and perhaps even the legitimacy) of the school’s administration-driven efforts. After the second year of this approach, members of the faculty asked for greater involvement in assess- ment—mainly because they began to realize the potentially significant ramifications of assess- ment for how they taught and wanted to keep an eye on and have input into an unavoidable task of which most remained skeptical.
A number of faculty members questioned the value of schoolwide assessment with argu- ments that fell into four categories: Institutional assessment of SLOs is not necessary, because each faculty member assesses student learning by assigning grades; institutional assessment of SLOs would be harmful, because it would lead to “teaching to the test”; this assessment cannot be done in a meaningful way, because what faculty teach cannot be quantified or evaluated objec- tively; and, finally, the effort would entail a large and unjustifiable drain on faculty resources and take faculty away from the important work of teaching, scholarship, and service. Lurking in the background (and sometimes expressed directly) was the suspicion that outcomes assessment was a vehicle for the University to exert control over the law school and therefore something to be resisted. But not all faculty members were so negatively inclined; indeed, a number embraced the idea of assessing student outcomes. They stressed the importance of identifying and stating learning objectives to ensure a clear insti- tutional sense of mission and direction and noted the relationship of assessment to what and how faculty teach.
Prompted by the faculty’s desire for greater input, the dean appointed the ad hoc faculty Committee on the Assessment of Student Learning Outcomes (CASLO) and charged it with developing, with faculty input, a proposed general approach to guide the law school’s assess- ment process and proposing how responsibility for carrying out these efforts should be shared in the future between the school’s faculty and administration. The committee’s work did not get off to a promising start: The chair was both wary of the potential for top-down assessment to interfere with meaningful instruction and concerned about the drain on faculty time and resources. But a funny thing happened over the course of that year: The chair, then the commit- tee, and finally a critical number of other faculty members came to appreciate the value of iden- tifying and assessing student learning outcomes, and the faculty as a whole adopted a sound set of principles to guide the school’s efforts. By the end of the third academic year with assessment, the ad hoc CASLO recommended, and the fac- ulty adopted, a number of important proposals, including the establishment of a standing faculty committee on assessment and the integration of an assessment process into the law school’s cur- riculum. CASLO went on to develop a list of the specific components that constitute a particular SLO. These elaborated SLOs help the faculty to
design and implement assessment; help students to understand the specific knowledge, skills, and attri- butes they should strive to achieve; guide the school’s development of curricular and cocurricular pro- grams; and help individual faculty members to identify and articulate learning objectives for courses. Developing a Culture of Assessment in the Dietrich School of Arts and Sciences
The Dietrich School of Arts and Sciences took a different approach for a couple of reasons. It would not have been feasible to attempt to centrally conduct significant portions of the assessment of student learning in 67 undergraduate and 60 gradu- ate programs, and the culture of the Dietrich School would have been even less welcoming to a top-down approach than the law faculty were. From early on, the dean of the Dietrich School was a champion of the institutional efforts to assess student learning because of the benefits he perceived for the academic programs. He cau- tiously but purposefully led his faculty over the course of four years to develop a comprehensive program of ongoing assessment of student learn- ing outcomes.
Recognizing the need for faculty buy-in, the dean made great efforts to ensure that this was not viewed as a meaningless exercise dictated by either the central administration or the Middle States Commission on Higher Education. With this in mind, the dean set out to engage the faculty in meaningful discussions of assessment of student learning, focusing on this process as a means to advance academic excellence and as a natural extension of the existing structure of program evaluation and the faculty’s interest in and ownership of the curriculum and its development. This framing of the project can be seen in the initial letters sent to the departments requesting that they identify the following:
59
university of Pittsburgh self-study rePort 2012
• What attributes, skills, and knowledge do you expect graduates in your major(s) to acquire that are characteristic of the discipline?
• What attributes, skills, and knowledge do you expect graduates in your major(s) to acquire that are hallmarks of your program at Pitt?
• What qualitative and/or quantitative evi- dence can you collect on an ongoing basis to show how well your graduates are meeting these goals?
In addition, departments were asked to identify how they would determine whether students were achieving those goals. The dean’s office was persistent in following up with
departments that did not submit responses while providing thoughtful feedback to those that did. The faculty soon realized that the assessment of student learning could not be ignored.
While setting clear expectations for each individual program, the dean also took purposeful steps to ensure that the school developed the appropriate culture of assessment. Initially, pro- gram plans were reviewed by the associate deans, but over time, members of the school’s curricu- lum committees became involved. Assessment of student learning was a featured topic at every chair’s meeting; at first, these involved presenta- tions by the dean and his associate deans, but after the first round of assessment plans was developed, chairs of departments with successful programs were asked to participate in panels to share their approach and their successes with colleagues. The annual Board of Visitors (BOV) meetings (attended by department chairs) regu- larly featured presentations about the culture of assessment developing within the school. In addi- tion, at each BOV meeting, a single department chair was asked to make a presentation about the department; for the past four years, each of these presenters has deliberately featured assessment as a central message in his or her report.