Program Evaluation Methods

The program evaluation of SBRPs is principally based on the external expert evaluation in the form of document-reviewing and panel review. The Research Area evaluation of SBRPs is regarded as program evaluations (Figure 4-4).

35

Figure 4-4. SBRPs R&D Management Evaluation System

The ex-ante program evaluation is performed by the Program Director Committee (Table 4-1) for JST to design Research Areas/Topics with types of programs and designates Program Officers, i.e., Research Supervisors to achieve each Strategic Objective/Research Theme.

Table 4-1. List of members of SBRPs Program Director Committee

Name Affiliations

Chairperson Shigeo Koyasu (M) Executive Director, RIKEN

Mitsuo Sawamoto (M) Professor, Chubu University

Atsuko Tsuji (F) Science Journalist, Specially Appointed Professor, Chubu University

Jun-ichi Tsujii (M) Director, Artificial Intelligence Research Center, National Institute of Advanced Industrial Science and Technology (AIST)

Kazuhiko Toyama (M) Managing Partner, Industrial Growth Platform, Inc. Yoshiro Hirayama (M) Professor, Tohoku University

Kazuo Hotate (M) President, Toyota Technological Institute Note: As of June 2020. Source: JST-SBRPs website.14

In the case of ex-ante program evaluation in FY 2019, the procedures were as follows.15

1) A preliminary survey based on the MEXT’s review of the Strategic Objectives

14 _{https://www.jst.go.jp/kisoken/en/about/index.html, [Last Accessed: 2020/5/10].}

36

(a) MEXT reviewed the Strategic Objectives in accordance with the guidelines for strategic objective formulation.

(b) JST received information from MEXT on the review stage stated above (in October 2019). JST conducted a preliminary survey to identify research areas that will contribute to the creation of innovation to achieve the Strategic Objectives.

(c) The preliminary survey of JST was conducted using the following methods:

- Based on the aforementioned review by MEXT, JST collected and analyzed information on research/ technology trends in the related fields and the status of relevant academic societies. - JST performed interviews with experts in related research fields, summarized and analyzed the

findings.

* 55 JST staff conducted interviews with a total of 232 external experts.

* Based on the relevant science and technology fields from the above reports, we used the following databases to overview researchers in Japan and selected the target interviewees: - Overview of researchers in Japan using

J-GLOBAL: A collaboration service of science and technology information operated by JST, which contains about 300,000 domestic researchers and 52.2 million bibliographic information on domestic and international literature;

JDream III: _{Japan's largest database of scientific and technological literature run by JST,} with approximately 70 million entries;

Web of Science: scholarly literature citation database provided by Clarivate Analytics, which contains about 20,000 journals

- Confirmation of past funding status and activities as an evaluator, stored in JST's internal Funding Management Database (FMDB).

(d) The progress of a preliminary survey continued to be reported to the Program Director in charge of each Strategic Objective for further discussion.

(e) Following the formal notification of the Strategic Objectives from the MEXT on March 9, 2020, JST proceeded with the further survey on designing new Research Areas.

2) Ex-ante evaluation and determination of the new Research Areas and Program Officers (a) The Program Director Committee was held on March 27, 2020, to conduct an ex-ante evaluation of new Research Areas and Program Officers.

(b) Based on the results of the above ex-ante evaluation, JST determined the new Research Areas and Program Officers on April 13, 2020.16

The interim/ex-post/follow-ups program evaluation is performed based on each Research Area by four to six external experts and specialists appointed by JST by avoiding Conflict of Interests. The program evaluation has two steps: the first step is the document reviewing by the panel member, and the second part is the interview-based panel reviews with Research Supervisor in charge except for the follow-ups program evaluation. The evaluation report is composed of the four-grade ratings, i.e.

37

Excellent, Good, Moderate, Not Good, and comments, except that the follow-ups evaluation is only for comment. Currently, JST equally conducted the interim/ex-post/follow-ups program evaluation in all Research Areas, i.e., about 60 Research Areas, without extracting any Research Areas to be evaluated in an intended or random way. Similarly, JST conducted the interim/ex-post/follow-ups research project evaluations in all research projects (Figure 4-5 and Figure 4-6).

Figure 4-5. Interim/Ex-post/Follow-ups Research Area Evaluation structure of SBRPs

Figure 4-6. Frequency and Targets of evaluation of SBRPs research areas and research projects

As mentioned above, JST has tried to capture the expansions and disseminations of achievements after the program and project as the follow-up program evaluation combined with the follow-up project evaluation since FY 2004. JST has developed the system for conducting a questionnaire-based follow-up survey in the 1st _{and 3}rd _{year after the projects and intensive document and interview-based}

38

follow-up survey in the 5th _{year after the program and project. These data and follow-ups evaluation} reports are openly published on the JST-SBRPs website and are also currently integrated into JST internal database. As these data have not been sufficiently analyzed, JST recognizes the challenges on how these data and results of follow-up evaluation will be adequately utilized as references to new program design and development (Figure 4-7).

Figure 4-7. Diffusions of Research Achievements using follow-up survey and follow-up evaluation