Table 4-2 shows the challenges and issues from the aforementioned program evaluations by FAs in major countries. With reference to the experience in the JST, the current issues of the research area evaluation are as follows.
1: Current Issues in the Research Area Evaluation
The Evaluation Committee comprises 4-6 experts without any special interest in the research area established for each research area (project) at the fiscal year being evaluated. More than 20 Evaluation Committees are set up annually. The members of the Committee review the evaluation materials and interview with each Research Supervisor (only written evaluation is conducted in the follow-up evaluation). A considerable amount of effort, resources, and costs are spent on the selection of experts who have no special interest in the research area and on the preparation of evaluation materials by the Research Supervisors, etc. (in the follow-up evaluation, these works are done by the research evaluation group and outside contractors).
In individual research area evaluation, the status of research area management and the achievement of strategic objectives are evaluated, but the appropriateness of the strategic objectives and the research area setting has not been evaluated. In addition, there is no mechanism to provide feedback to MEXT on its recommendations. As a result, it cannot be said that the PDCA cycle is necessarily being implemented, and the ex-post Evaluation Committee members, in particular, have pointed out problems with the use of the results. In addition, the ratings of the evaluation report (four-point scale of A+, A, B, and C) remain too high (no B or C rating was given in recent years), and the evaluating members tend to be indulgent in their evaluation of the research area.
The hypothetical proposal of a pathway to outcomes and impacts toward the achievement of the strategic objectives and research area goals assumed by the research area at the ex-ante evaluation is not explicitly validated at the time of interim and ex-post evaluations as described in the general guidelines for national R&D evaluation. In addition, the hypothetical proposal is not modeled (such as a logic model).
To address the above issues, we propose the following directions for improvement of the research area evaluation process. The timing of the transition to the new system is targeted to be from the fiscal year when the next medium and long-term plan begins. However, we will start some items such as the
47
improvement of efficiency of follow-up surveys, if possible, from the next fiscal year.
2: Directions for Improvement
(1) Larger Groupings of the Evaluation Committee
The Evaluation Committees that have been set up by research areas will be grouped together by strategic priorities fields (e.g., life innovation, green innovation, nanotechnology and materials, ICT) (see Figure.4-12).
Considering the specialization and diversity within each field, the composition of the committee members in each field is expected to be about 8-10 members, including external experts who are not full-time members but are commissioned for each research area (the current number of Evaluation Committee members for each research area are 4-6). The term will be two years (renewable). In addition, given that the Program Directors will conduct the ex-ante valuation of the research areas, it may be effective to ask them to attend the committee meetings and to exchange opinions with the Research Supervisors and the committee members.
The new field-specific Evaluation Committee for each strategic priority field will collectively conduct cross-cutting evaluation across the research areas of which interim/ex-post evaluations will be conducted in each evaluation year. However, depending on the number of research areas to be evaluated, the Evaluation Committee may consider splitting the committee meeting into several times.
(2) Role of the New Evaluation Committee
The new field-specific Evaluation Committee will discuss and evaluate the appropriateness of the research area setting at the interim/ex-post evaluation of the research areas, as well as the status of research management and the achievement of strategic objectives. This will result in a more formative evaluation that emphasizes improvement and learning than in the past. If the content of evaluation reports by the field-specific Evaluation Committees will be changed from summative to formative (emphasizing dialogue between Research Supervisors and committee members, management improvement and learning), consideration should be given to the possibility of a two-level evaluation, for example, to determine whether or not the research has achieved results that contribute to the achievement of strategic objectives.
In addition, we will ask for initiatives, developments, and recommendations that can be useful for setting up new research areas in the ex-post research area evaluation. We will also ask for general comments on the field as a whole, as well as recommendations for program improvement and strategic objectives. We will consider a mechanism to share the evaluation results and recommendations of the field-specific Evaluation Committee with MEXT and utilize them for future improvements, etc.
In order to reduce the burden on each Evaluation Committee Chair, it may be possible to appoint one or two vice-chair(s) to share in the preparation of the report.
The evaluation materials prepared for each research area will be simplified as much as possible, taking into account the intentions of Evaluation Committee members and Research Supervisors. (3) Promotion of More Efficient Follow-Up Survey/Evaluation and Impact Survey
48
Regarding the follow-up evaluations, the evaluation of each individual research area (project) will be abolished. As an alternative measure, the results of the follow-up surveys conducted over multiple years will be analyzed and evaluated by the research evaluation group, and the field- specific Evaluation Committee will collectively evaluate the results once every few years (Consideration will be given to not conducting follow-up evaluations again in the year when the International Review is conducted (every five years)). In this case, the follow-up survey will be streamlined by optimizing the data collection necessary for the survey and reducing costs. For this purpose, the research evaluation group will conduct tasks that can be done by itself, including researcher interviews, data analysis and evaluation, and preparation of materials, as the minimum required for data collection.
The reduced amount of resources and costs by improving the efficiency of follow-up surveys and evaluations will be allocated for extracting outstanding results generally ten years after the end of the research area and for effectively surveying, analyzing, evaluating and publicizing their impact. This impact survey will be used as a source of evidence to show the effectiveness of the investment in basic research and to analyze the ways how the results of basic research create impacts (see Figure 4-13).
(4) Strengthening Bridging Activities
As a review of the work content and activities of the research evaluation group, we will divert resources from evaluation activities to strengthening bridging activities that will lead to the identification of promising technology seeds and further development of results. In line with this, we will share and utilize data within and between relevant sections and departments to share the information of technology seeds in conjunction with the next Funding Data (FD) project. We will review the timing and implementation methods of the periodic survey conducted one
year and three years after the end of the project for use as a follow-up. At the same time, the survey method will be modified and the results of the previous survey will be reused to ease the burden of answering the questionnaire, including the follow-up survey conducted five years after the completion of the research area/project, to the extent possible.
(5) Others
As SBRPs are focused on basic research, which might not be linear to the outcomes, it is not appropriate to uniformly apply the logic model (a pathway to outcomes and impacts toward the achievement of the strategic objectives and research area goals) at the ex-ante evaluation phase to all research areas. However, there may be cases where the logic model can be used depending on the characteristics of the research areas, so we will continue to examine its applicability and usefulness.
49
Figure 4-12. Larger groupings of program evaluation from Research Areas to Strategic Priorities Fields
Figure 4-13. Impact Assessment Framework for ex-post projects