The surveys conducted for this dissertation aim to provide results that will contribute to understanding how decision-making and change in institutions of higher education may be influenced by the participation in and results of national
assessment studies such as the NRC. The literature reviewed in Chapter 2 provides evidence that the implementation of sustainable and meaningful change is difficult and requires certain themes to be present, such as key champions, institutional leadership, and accepted priorities toward continuous improvement. The survey results assist in showcasing campus reactions to the NRC study as one example. Analyses of the survey results show meaningful outcomes: while there is clear uncertainly about the validity of the NRC, many examples exist of universities and academic programs using the study data and results to assess their performance and improve themselves.
Results of the seven surveys discussed in Chapter 3 were analyzed with tools in the Qualtrics software. Data were also downloaded into CSV files for analysis in STATA and Microsoft Excel software. Presented below are results and discussion of the central administration survey, all six program level surveys combined, and several examples of individual program survey findings that were especially
noteworthy. Detailed tables with descriptive statistics and frequencies from each of the surveys can be found in Appendices 4.1 through 4.9. A series of tables showing
the results from the central administration survey can be found in Appendix 4.1; the combined results from the six program-level surveys can be found in Appendix 4.2; the combined results from the high-quality program-level surveys can be found in
Appendix 4.3; and the results from each of the disciplinary surveys can be found in the appendices following in order. The analysis will refer to these tables.
Central Administration Survey
The central administration survey was distributed in September 2011. There were a total of 104 useable surveys for a 50% response rate. On the question of whether the NRC study achieved its stated goals, 64% of respondents (n=67)
reported it did not. At the same time, however, 52% of respondents (n=54) reported the NRC results have been incorporated, or there were plans to incorporate, in campus activities or discussions.
Responses from Table A4.1-1 show that the NRC results are used generally in areas of assessment and evaluation (e.g., general conversations about graduate education with n=39 and 75% of question respondents reporting this use, or program review with n=36 and 69% of question respondents reporting this use) more so than in areas of action or decision-making (e.g., budget or resource allocations with n=15 or 29% of question respondents reporting this use, or advocacy to state
governments with n=5 and 10% of question respondents reporting this use).
Program review and discussions about the quality of graduate education on campus are more evaluative in nature than advocacy activities, policy changes, or
determining areas of growth for new graduate programs. These findings suggest the NRC results are used as a tool for considering the quality and future assessment of
graduate programs yet not relied upon heavily for decision-making on campuses. These findings are consistent with the survey open-ended comments regarding the use of the NRC study on campuses, as well as the findings discussed nationally since the results were released (NRC Convocation, 2011). This distinction from the survey was assessed further during the case study portion of this research.
The NRC Convocation on Analytic Uses in March 2011 provided evidence that universities found value in preparing, discussing, and collecting data for the NRC study. It was asserted that some universities found the preparation for the NRC study more valuable and meaningful than the results themselves. The central administration survey asked respondents about this idea, and the responses turned out to be very similar to the general use or plans for use of the NRC study on
campuses.
By way of example, Table A4.1-2 shows the difference in responses between the question about use of the results generally and the question about use and value of the study in advance of the results being released. The results show fewer
universities used the data for action or decision-making purposes prior to the study’s release (e.g., six fewer respondents reported use of the NRC study for budget or resource allocations and five fewer respondents reported use for advocacy purposes in advance of the release as compared to the general use question). Five more campuses reported holding retreats or discussions to discuss important issues in graduate education, likely an indicator of preparatory activities for the NRC study release. But the findings do not show that many more campuses incorporated the preliminary data into decision-making activities or included them in evaluative or
assessment exercises such as program review. This finding suggests that those campuses with robust assessment and decision-making activity wanted to wait to see the actual study results before incorporating them into their campus processes. The NRC Convocation presentations and debate may have been reflective of those campuses who were growing their institutional research and data assessment capacities.
Data collection and analysis are the two areas where survey results showed major differences between responses about general/planned use currently and the question that asked about any uses in advance of the results being released. For these two choices, 16 more institutions reported engaging in data collection and data analysis efforts with the NRC study in advance of the results being released as compared to once the results were public. Faced with the somewhat daunting task of providing vast amounts of data to the NRC about their campus resources, faculty, and doctoral programs, universities likely realized gaps in their data availability and knowledge. Campuses took steps to collect and analyze more data about their people and activities, knowing the NRC study was pending. The implementation of these practices, assuming they were not one-time data collection mandates,
suggests opportunity for sustainable change.
Universities could have used the NRC study definitions and data variables as a framework for the types of data they should collect on an annual basis, both to assess trends in these key national variables but also to prepare for the release of rankings on these variables. The release date of the NRC study results was pushed back several times. While exact numbers are not known, during this time, some
universities likely continued to collect data on their graduate programs. At the NRC Convocation, universities presented examples of new reporting efforts that
developed on their campuses as a result of the NRC study. In one sense, preparation for the release could have become a continuous process on some campuses. In an effort to collect data to augment or refute potential NRC study results, universities had the opportunity to create institutionalized tools for data collection and assessment.
The remaining questions in the central administration survey asked respondents to rate their agreement with several statements concerning the
usefulness of various aspects of the NRC study as well as factors contributing to its perceived usefulness. Tables A4.1-3 and A4.1-4 show these statements and the respondents’ assessment of them.
Responses show that the processes involved in participating in the NRC study are rated more useful (n=59 for 60% of respondents) than the rankings that resulted (n=46 for 46% of respondents). The information included in the study’s database/Excel spreadsheet was also found to be useful by 60% of respondents (n=61). Because there are few studies of this magnitude, preparing for the study, collecting the campus data, and subsequently having access to the data about all doctoral programs nationally appear to have contributed to the perceived usefulness of the NRC study. At the same time, the resulting rankings were not rated as highly useful as the database alone, a finding that introduces questions about the utility of this type of study, or at least the methodology and outcomes of the study.
A high number of respondents indicated their campuses had active
participation among faculty and staff preparing for and participating in the NRC study (n=67 for 67% of respondents), including “champions” for the study (n=75 for 76% of respondents). This finding is in line with use theory that active involvement and champions for a project increase its perceived and actual usefulness. Because of the complexity and breadth of the NRC study, both in scope and in time, without campus leadership and engagement, its level of impact and usefulness would be significantly dampened.
A third of respondents (n=31) reported that the NRC study will improve the quality of graduate education on their campuses, an important finding. However, far fewer chief academic officers reported that the results persuaded them to implement change on their campuses (n=17 for 17% of respondents). Conversely, 40% of respondents (n=40) did agree with the statement that the NRC study and results have or would be used to justify decisions. The dichotomy between persuasion and justification to change was studied further in the case studies as described in