Models and cases selection 1 Selection of the models for review

Different countries use different models to review research activities in universities. . International models approaches were favoured because of their more advanced approaches that laid a foundation for the study. International participation and international competence are also regarded as important aspects of developments in South Africa.

A purposeful selection of models (McMillan and Schumacher, 1993) was guided by an extensive review of sources on research evaluation and research quality assessment coupled with intermittent verbal verification with different scholars at international conferences and centres. The reason for the selection was guided by the systemic nature of HE in South Africa. That is, that model to be studies would simulate systemic patterns and provide information on how evaluation of research can contribute to the improvement of research in such a system in order to address imperatives of such a system. The Dutch model was selected for its least differential nature with planned intentions for institutional accountability necessary for the autonomous South African Universities while the UK model shows show a

different approach to this. Such information would guide the researcher on the choice of suitability of practices. Both models have a long standing history and are at advanced stages. The New Zealand model on the other hand is recent and it was hoped that with the history similar to that of South Africa, the model would reveal activities that are necessary for the local situation.

Some information had to be obtained prior to the final selection of models, to improve

reliability on the models selected. The exploration was then followed by in-depth studies of a few models. This type of sampling is more appropriate to multiple-case designs studies than the maximization of variability, even though comparative analysis could not be avoided. The study of models followed a historical approach in order to expose the evolutionary processes and to reveal contextual differences at the different levels of development before a

comparison could be drawn.

The choice of models was based on a ‘reputational’ approach informed by historic events and experiences accrued, especially at national level. While some countries have been observed to use pluralistic approaches to evaluation (Geuna and Martin, 2001), consideration was given to Campbell’s (Shapira and Kuhlmann, 2003) advice on the tendency of such approaches to evolve towards the systemic and comprehensive type when they develop. Countries that have been involved in research evaluation over a longer period of time, The Netherlands and the UK, stood better chances of being selected as they may be regarded as connoisseurs in the field (Campbell in Shapira and Kuhlmann, 2003, Lucas, 2004).

In both countries, research evaluation processes have been in existence since the early 1980s. All government subsidized (public) institutions participate in the exercise. This fitted with the intended focus of the study, South African public universities. The emergent model of New Zealand was included since, by context, it was perceived as being as close to the South Africa situation as possible. The New Zealand comprehensive/large-scale model is relatively new and is preferred as a guide for efforts to establish a new model.

Activities, practices and experiences of research evaluation prior to the introduction of the other countries’ new models are also reviewed in this study. This was done to minimize threats of reliability and validity that could occur as a result of the vast differences in experiences and contexts between the “connoisseurs” and South Africa. As previously

indicated, May’s (1997) advise not to set standards beyond context and the level of judgment was taken into consideration. This is the main reason why some information had to be obtained prior to the final selection of the models. Care was also taken to study and conceptualize the selected models without any desire to generalize or even extrapolate beyond the limits of application.

The number of models selected was influenced by the availability of relevant information based on the main purpose of the study. Therefore, documents of models that had the logical potential to yield relevant information guided the selection exercise. The Dutch model was preferred for its evolving purposes, all geared towards high quality research productivity across the entire system. This would guide the South African imperative to reduce disparities. The UK model on the other hand, was selected because of its historic similarities with the S.A. efforts (purpose for funding) It should also be noted that most of the existing South African research evaluation efforts refer to the UK and the New Zealand models. A possible exclusion of similar models (systemic and comprehensive), if any, may be viewed as a bias towards the selected models, and this, if it exists, creates a setback for this study.

In systemic/comprehensive/large-scale research evaluation, governments are directly or indirectly involved to ensure processes of accountability and improvement (Campbell in Shapira and Kuhlman, 2003). Intermediary organizations are usually formulated in

collaboration or in agreement with governments and assigned the responsibility to facilitate the process. In most cases if not all, the intermediary bodies engage advanced researchers. Examples of such organizations are the VSNU in The Netherlands, the HEFCs in the UK and the Performance-based Research Funding (PBRF) in New Zealand both of which are

government bodies/structures. The NRF is a local example of such intermediary bodies.

In the United States, Canada and Germany ‘federal” states operate autonomously with some guidance from the national department. In Canada, provincial operations have been reported whereby individual provinces may decide on how evaluation will be performed. In the US on the other hand, different constituencies are used for different purposes such as benchmarking and standards making the US system one of the most advanced differentiated higher

education research systems. The exclusion of these models does not in any way suggest that they are unsuccessful but rather that this study focused on research evaluation of a single system of HE placed under a single DHET. The German situation illustrates how the Lander

(regional) setup still influences the operation of universities (Geuna and Martin, 2001). The fact that the South African HE system is a unified one led to the exclusion of these federal models. Although South Africa may learn from the federal state the intention of this study was designed to fulfill national imperatives. Therefore systems such as those in the US, that are highly differentiated were not the focus of this study.

3.3.2 Selection of the South African cases.

Relevance to the study has been the determining factor for the selection of sub-cases within the South African case. This followed Yin’s (2009) embedded case study design, in order to accommodate comparisons between the South African Type B model with the other

countries’ Type A situation within the same phenomenon.

Figure 3.2 The multiple-case design with one case showing embedded cases (adopted from Yin (2009 ) and altered for this study

Efforts were made to identify bodies/structures in the country that are involved in research evaluation of universities in one way or another. For purposes of better understanding, the selected (participation) structures/bodies have been discussed in separate chapters in the study. Government documents and those of other parastatals involved in research evaluation exercises are mentioned below. Figure 3.2 depicts the embedded case study design as used in this study.

3.3.2.1 The Higher Education and related legislative documents

South African legislative documents relevant to HE research evaluation have been referred to in this study. To build on the history of South African research evaluation, documents of the old order (apartheid system) have been consulted. The emphasis, however, is on documents of the new system of higher education (from 1994) as indicated below:

- The White Paper 3 (DoE, 1997a) provides the conceptual framework for the

transformation of HE in South Africa and led to the development and adoption of the Higher Education Act 101 (DoE, 1997b), which regulates the system and provides the legislative framework. The National Plan for Higher Education (DoE, 2001) is an operational document providing mechanisms for implementing ideas of the White Paper 3 and the Higher Education Act 101. All these are important in the regulation of activities of the public HE system which is the focus of this study.

- The White Paper for Science and Technology (1996) on the other hand, has been included for its role ‘in shaping and steering research’ in the country.

3.3.2.2 The Council on Higher Education (CHE) documents

The CHE and its committee, the Higher Education Quality Committee (HEQC), have been established and are regulated by the Higher Education Act 101. The inclusion of the CHE documents in this study was informed by the fact that its mandate is to improve quality in HEIs. The Council has employed different approaches to manage the situation and those that relate to or are involved in research evaluation have been included in this study. Those are:

- The Founding Document (2001)

This has been included to provide background information about activities of the Council.

- The Framework for Monitoring and Evaluation in Higher Education (2004) In terms of legislation, the CHE is required to advise the Ministry of Education on the performance of HEIs for the purposes of efficient funding. Research is one of the components of institutions funded by the Department.

- The CHE Framework for Institutional Audits (2004)

The CHE, through its committee, the HEQC, audits the performance of institutions for quality purposes. These audits provide information to the Minister of Education. Although

one of the functions of the Audit wing is the accreditation of programmes, its involvement in internal quality management of institutions makes it relevant to this study.

- A Good Practice Guide for Quality Management of Research (2005)

This document of the CHE is closest to the aims of this study in that it focuses on HEIs research activities nationally except that developmental responsibilities are left to individual universities meaning that the stated activities in the Guide do not constitute a national

research evaluation exercise. It’s value and practices have nonetheless been studied to guide the current study’s recommendations.

3.3.2.3 The Department of Education document (2003)

- The Policy and Procedures for Measurement of Research Output for Public Higher Education Institutions

This is the DoE’s system of rewarding research productivity. Its direct involvement in research evaluation informed its inclusion in this study.

3.3.2.4 National Research Foundation (NRF) involvement

- The National Research Foundation Rating System (NRF 2002)

The NRF is an intermediary (parastatal) establishment responsible for developing research in South Africa. The rating system was intended to promote and support excellence in research and to support curiosity-driven research. The level of excellence has in recent years been used to encourage national and international competitiveness, an important transformation imperative.

The NRF has also been identified by the Department of Science and Technology as a body that should assist in the process of unifying research activities and the evaluation thereof.