INFORMATICS EUROPE ZURICH 9 JUILLET 2019
EVALUATION and OPEN SCIENCE
Three little quotes
Goodhart's law:
"When a measure becomes a target, it ceases to be a good measure.” (Wikipedia)
Campbell's law:
"The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor." (Wikipedia)
David Jones AKA Daedalus:
“(…) once performance indicators are taken seriously, doom is certain. The organization switches its goals from whatever it should be doing, to
CNRS
Snapshot
The National Centre for Scientific Research is an interdisciplinary public research organisation under the administrative supervision of the French Ministry of Higher Education and Research.
— € 3.3 billion annual budget
— 33,000 people dedicated to research
— ± 10,000 permanent researchers
— 1,144 research laboratories in France and abroad
— ±1,000 are joint with (mostly) universities
— 10 “Institutes”:
• Biology
• Chemistry
• Ecology & Environment • Humanities & Social Sciences
• Information Sciences and Technologies
• Engineering and Systems Sciences
• Mathematical Sciences and their Interactions (INSMI)
• Physics
• National Institute of Nuclear and Particle Physics (IN2P3) • National Institute for Earth Sciences and Astronomy (INSU)
An open science policy
“Open science is just science done right” (Jon Tennant)
Infrastructures (20 years of development) Revues.org (1999, now Open Edition)
HAL/CCSD (2001)…
Was this thought as “open science”? Was “LANL preprint archive”, ancestor of arXiv, an open science
initiative? No! just science done right!
Episciences… Peer Community in X (PCI)…
To cite two among Many many initiatives…
National coordination
In the framework of the national open science policy International cooperation
COAR, Knowledge Exchange, OpenAire, Operas, Hirmeos, EOSC, Etc.
A road map at CNRS
Evaluation as the main impediment
A double bind:
Be open! Say the policies
But you soon hear that your peers will evaluate you on your publication record in prestigious venues
And two aspects about the second premise:
Prestigious venues are not open…
CNRS has signed DORA
(San Francisco Declaration on Research Assessment)
Among the 17 recommendations, two are specifically for RPOs:
4. Be explicit about the criteria used to reach hiring, tenure, and promotion decisions, clearly highlighting, especially for early-stage investigators, that the scientific content of a
paper is much more important than publication metrics or the identity of the journal
in which it was published.
5. For the purposes of research assessment, consider the value and impact of all
research outputs (including datasets and software) in addition to research publications,
and consider a broad range of impact measures including qualitative indicators of research impact, such as influence on policy and practice.
Comité National de la recherche scientifique:
Evaluation of researchers at the CNRS
The National Committee for Scientific Research (CoNRS) is a collective body comprising the Scientific Board, Institute Scientific
Boards, specialized sections in each discipline, and Interdisciplinary Commissions. The CoNRS plays a key
role in French science.
Its members are representatives of the scientific community who apply their expertise to support the core mandate of CNRS (…) Thanks to the work of its governing bodies which manage around 20,000 files each year, CoNRS helps define the scientific policy of the CNRS, analyze the current situation and outlook, recruit researchers and promote their careers, as well as
monitor research unit activity.
41 disciplinary “sections” and 5 interdisciplinary “commissions”
Each composed of about 20 Peers.
Each defines and publish its own evaluation criteria…
Common criteria: 1/4 principles
It is only the scientific results that must be evaluated, not merely the fact that they may have been published in a prestigious journal or other renowned media:
The members of the National Committee for Scientific Research must take up the responsibility of their own appraisal and will not rely on anonymous
publisher evaluations or algorithms. This should be made apparent in the evaluation reports..
Common criteria: 2/4 principles
For each of the research output listed in their evaluation dossiers, researchers must explain its scope, impact, and their personal contribution to it:
Common criteria: 3/4 principles
All types* of outputs must be eligible for evaluation:
In particular, whenever appropriate, the data underlying the publication and the source code necessary to produce the results must be made available. “Pre-prints” and other working documents are acceptable materials for evaluation. The same applies to “data papers”.
*See e.g. HCERES guides for research products:
Common criteria: 4/4 principles
Whenever possible*, all outputs listed in the evaluation dossiers must be accessible in HAL or alternatively in another open archive:
The relevant material is the outputs themselves and not their citation. It is normally not necessary to provide them in the folder: only the active link to the archive should be adequate.
*Three exceptions are acceptable:
1. Recent outputs falling outside thescope of the 2016 Law. A private URL should then be provided.
2. For recruitments, this rule will not apply strictly for candidates coming from abroad or from private institutions. 3. The output type is not eligible for HAL.
What is to be done?
Что д лать?
• Incorporate the four principles as a preamble into each of the 46 sets
of criteria, possibly with some adaptation to disciplinary specificities • Amend each 46 texts…
