Marie Curie Fellowships - Guided Tour - Tour 6 - Peer Review System

Marie Curie Fellowships

TMR* Fellowship Peer Review System

^* TMR (Training and Mobility of Researchers Programme) is the predecessor of the Human Potential Programme. This article refers to the peer review system used in the selection of the Marie Curie Fellowships during the Fourth Framework Programme. This system will continue to be used in the Fifth Framework Programme.

(Published in the ESTA News Nr 3 - Spring 1997)

Introduction

The scientific evaluation of research programmes inevitably involves a peer review procedure. While the objectives of most peer review schemes are similar (fairness, objectivity, consistency and transparency) their form varies considerably.

The quality of any research programme is strongly dependent on the quality and integrity of its accompanying peer review procedure. The prestige and standing of the research programme also depends on the acceptability/reputation of the peer review process.

TMR Fellowship Programme

The TMR Programme operates trans-national fellowship schemes at postgraduate, postdoctoral and also at senior scientist/visiting professor levels.

The fellowships are provided for a maximum of 3 years at postgraduate level, 2 years at postdoctoral level and 1 year for senior scientists/visiting professors. The programme covers all fields of the exact, natural, economic and management sciences as well as those social and human sciences that contribute to the objectives of the Fourth Framework Programme (industrial competitiveness and quality of life).

All Member States of the EU participate together with Norway, Liechtenstein, Iceland and Israel.

Peer Review Procedure

The scientific evaluation of the fellowship proposals are reviewed in 7 panels by independent experts nominated by national governments of the participating countries, European scientific bodies and industrial associations. In appointing evaluators, a reasonable distribution of expertise between the various sub-disciplines is attempted. Attention is also paid to the balance between countries and between representatives of academia, research centres and industry. The size of the panels depends on the volume of applications in a particular discipline, which are currently as follows:

These experts were selected from a database of over 4000 and are appointed for a maximum of 3 years. A minimum of 1/3 of the panel members are rotated every year. The panels are chaired by one of the scientific experts who is appointed by the services of the Commission. The chairman plays a key role in the rotation of the panel members.

The entire peer review procedure takes place in Brussels over a 2-3 day period. The panels are assisted by a panel secretary and a Commission representative (staff members of the TMR Programme). The role of the Commission representative is to ensure that the rules and regulations concerning the peer review procedures are correctly followed. It is important to note that the staff of the TMR Programme are not themselves involved in the scientific evaluation of proposals.

Each proposal is first allocated to a particular panel, according to the wishes of the applicant. The project is then passed to 2-3 experts for individual evaluation. This evaluation takes into account the project classification by sub-discipline as indicated by the applicant. Procedures also exist for the handling of multi-disciplinary or cross-disciplinary applications. Panel members are provided with a list of proposal titles prior to the panel meeting and are asked to indicate projects that they would be interested in evaluating. These "indications of interest" are used by the panel chairman/panel secretary as input into the assignment of projects to evaluators. The chairman of the panel, assisted by the panel secretary, decides on the final allocation of projects. However, each expert, having been provided with the projects at the panel meeting, must confirm that they consider themselves competent to evaluate such a proposal. At this stage, experts also confirm that no direct or indirect conflict of interest might arise from their evaluation of the individual projects allocated to them.

Proposals are evaluated principally on the grounds of scientific excellence; as the TMR Programme is a training through research activity, training aspects of the proposals are very important.

Each expert gives a maximum of 100 points to a proposal according to the following: quality of project - 30 points, appropriateness of host institutes in providing training - 30 points, quality of applicant - 40 points.

Detailed information is contained in a document "Guide to Evaluation and Selection of Training through Research Proposals". This document is available directly through the services of the Commission and on the Internet: /tmr/home.html.

The consistency of marking of experts is monitored during the panels and techniques are used to identify high, low and narrow (truncated) markers. Systematic high and low markers are identified using an analysis that compares the score of an expert against his or her 25 or so partners. Narrow markers are identified by the expert's score distribution patterns through the calculation of mean and standard deviations. The frequency that an evaluator's name appears in the list of recommended projects and in the lowest 15% of projects is also monitored. These analyses may be supplemented by scores allocated by the expert in previous panels (historical data). Such techniques are applied early in the peer review process in order to eliminate inconsistent score variation and as an aid to help reconcile differences between evaluators. Any difference in marking between experts that is greater than 10% is referred to the chairman who may then assign the project to additional evaluators. It was found that in those panels where score monitoring was practised the need for additional evaluators significantly decreased. This has led to the routine use of the monitoring techniques in all panels.

After the individual evaluations are completed, each panel collectively discusses the results, particularly of those projects that required additional evaluation. Particular attention is paid to the analysis of the evaluators' marking patterns. Finally, the panel agrees on the list of projects recommended for funding and, in the event of applicants withdrawing or refusing an offer of funding, a reserve list. At no stage are "national quota" taken into account in the evaluation and funding of applications.

A prominent independent scientist (not a member of the panel) acts as an observer and reports to the Programme Committee (committee of national representatives that guides the TMR Programme).

Summary

The described peer review system allows the efficient evaluation of 2500 fellowship proposals every six months. The introduction of techniques to monitor the scoring pattern of evaluators has significantly contributed to the fairness and consistency of the peer review process.