Access to Research Infrastructures - How to apply - Report Independent Observer

Access to Research Infrastructures (ARI)

Enhancing Access to Research Infrastructures Third Call 2000
(Call Identifier IHP-INF-00-1)

Report of the Independent Observer of the Assessment of Proposals
Katharine Barker, Victoria University of Manchester

1. Summary of Main Findings

• Overall, the evaluation procedure was efficient, well-informed and fair, with the overriding emphasis being upon the scientific quality of the proposals. This is a considerable achievement given that peer review does not typically work well across disciplines. The evaluation followed the criteria and guidelines set out by the Commission.

• The change in procedure since the first call, which allowed discussion of each proposal within smaller groups, was strongly endorsed by the evaluators.

• The evaluation panel was well balanced in terms of expertise and country of origin.

• The Commission provided excellent briefing documentation and presentations. The Chair performed a key role in organising the meeting.

• The evaluation process was straightforward, with primary assessments made by individuals being the basis for discussion and arrival of a common view within the groups, and ranking of the selected proposals at a plenary meeting.

• Allowing the evaluators to make a primary assessment of proposals at their normal place of work is a major advantage and contributes significantly to the quality of the evaluation process.

• The decisions of the evaluators were based upon discussions of scientific quality and budgets required, and were taken in a highly professional way. The feedback given to proposers was carefully considered within the groups, by the group rapporteurs and Chair of the evaluation panel.

• The criteria for scoring were found to be workable and appropriate, although a greater weighting on criteria relating to scientific quality might be appropriate for this programme.

• The rules on conflict of interest for evaluators are appropriate, except in the case of nationality. In a basic science programme, national interests are not relevant for expert evaluators.

2. Suggestions for Future Evaluations

• The system of convening smaller groups with a rapporteur should continue and be strengthened with explicit guidelines for the non-specialist rapporteur. The guidelines should stress the need for the rapporteurs to ensure consistency between marks awarded and comments given by evaluators. Also, the rapporteur should actively seek consensus and discussion from the all the members of the subgroup.

• The procedure for ranking the final list of selected proposals could be recommended to future panels.

• The programme's practice of mixing evaluators (some with experience of the programme and some new to the programme) is valuable and should continue.

• While information from mid-term reviews is available for evaluators, it could be provided for each infrastructure to accompany the proposals.

• Information for proposers should stress that costings should be very clearly elaborated and that the scientific quality of the infrastructure is extremely important.

• If much larger bids for EU research funding become a feature of future programmes, then evaluation should include experts from outside Europe and may require more interaction with proposers (such as visits and/or interviews).

3. Introduction

This report comprises the observations and advice of an independent person on the conduct of the evaluation of proposals for the third call (2000) for proposals under the theme "Enhancing Access to Research Infrastructures". It has been produced in accordance with the terms of reference given in Annex II of the Guide for Evaluators for the call. Thus, the independent expert was provided with the briefing documentation for the evaluators and attended both the briefing meeting for evaluators and the evaluation panel meeting (attending the working groups and the plenary sessions). At the end of the panel meeting the experts gave their views on the process in plenary session. During and after the meetings she held informal discussion with the evaluators and with Commission staff. She observed the entire evaluation process and sat in on the meetings of all the sub-groups as well as the plenary sessions.

The report comments upon the different stages of the evaluation process, makes some reflections upon it and suggests actions for the future.

4. The Evaluators

The 51 evaluators were from all countries of the EU and also Switzerland, Norway and Iceland. 14 were women. 5 were from industry. The Commission staff had to assemble the panel from the database of evaluators before the profile of the proposals was known, which is not ideal, particularly as the programme is "bottom up" in nature and can attract proposals from any area of research. The panel turned out to be well-balanced in terms of the mix of disciplines required for the 149 eligible proposals received Some had experience of the programme (through participation in mid-term reviews and/or previous evaluation panels) and others were new to it.

5. Overview of Evaluation Procedure

The evaluation process was fairly straightforward, considering the large size of the call and the variety of instruments available to proposers. Following the briefing meeting at which the evaluators allocated the proposals according to expertise (with three evaluators per proposal), the experts examined the proposals in their normal place of work. They filled in the assessment forms and sent them electronically to the secretary of the panel.

At the second meeting, each proposal and the primary assessments of it were discussed by subject area groups of between 7 and 16 evaluators with a rapporteur from outside that subject area. The subgroups agreed upon final scores, a ranking order and the form of feedback to the proposers. A plenary meeting on the third day ranked the proposals and produced a final priority list of proposals accepted for funding.

6. Briefing Documentation

The Guide for Evaluators for the call was sent in advance (along with the proposal forms and Guide for Proposers). It gave full explanation of the stages of the handling of the proposals, the evaluation criteria (including weighting and thresholds) and the assessment forms to be used. It also explained the Commission rules on conflicts of interest. The documentation was also available on the web. It was clearly written and comprehensive.

7. Briefing Meeting and Allocation of Proposals

At the briefing meeting, Commission staff gave a talk explaining the context of the evaluation and its relation to the previous call and to the Fourth Framework Programme contracts. The secretary to the evaluation panel provided an explanation of the evaluation procedure, criteria and scoring, stressing the rules on confidentiality, conflicts of interest and how to complete the forms. The presentations by the Commission staff were detailed and informative and allowed the panel members to raise several questions about the procedure. Although some of the information was available in the Guide for Evaluators, it was very helpful to have this panel meeting to ensure that the evaluators were perfectly clear about their task and the way of organising the assessment.

Guided by the Chair, the evaluators assigned themselves to about 6 or 7 proposals per person. The evaluators declared interests in proposals, according to a very broad definition, which included any proposals from the same institution, proposals which might directly or indirectly be in competition with their own activities and proposals from facilities which they used. In addition, experts did not evaluate proposals where the proposal coordinator shared their nationality. The discussion of proposal allocation also allowed the category of the proposal to be checked; in a few cases proposals were re-allocated. Despite these rather stringent conditions, the allocation of proposals proceeded fairly easily. The evaluators agreed on their participation in one or two of the subgroups. After the meeting some "orphan" proposals were allocated by email. While it was not possible for every evaluator to have a precise match in expertise with the content of all their allocated proposals, the match was good enough, and having three experts per proposal was a safeguard to ensuring fair consideration of all bids.

The rules of confidentiality were stressed by the Commission staff. Where additional expertise was necessary, the evaluators were asked to consult with colleagues under the same conditions of confidentiality.

8 evaluators were unable to attend the briefing meeting.

8. The Primary Assessments

During the period between the briefing meeting and the evaluation meeting, the proposals had been assessed. The Commission staff had assembled the primary marks and comments and also information about proposals which had previously been rejected or funded in the past. There were some difficulties in using the numbering system and filling in the forms, but these were all resolved by email in time for the panel's second meeting.

Commission staff provided additional information with the proposals: comments of the earlier evaluation panel sent to the applicants in the case of resubmission, and information on costings for existing access contracts. Evaluators were invited to look at the web pages for the results of mid term reviews. In one case (a network proposal) the information about an existing network contract was not supplied and the evaluation process took rather longer than it otherwise might. This illustrates the value which evaluators place upon the additional information, particularly about past performance.

9. The Panel Meeting - Workings of the Subgroups

All the evaluators attended the panel meeting (except one, due to illness), which took place over 3 days. First, a plenary meeting established the working of the subgroups. The Commission staff answered questions about conflicts of interest and the duration of contracts.

Led by the rapporteur, each subgroup considered proposals in turn for discussion, for about 15 to 20 minutes (in some cases, longer). The approach was not to take a simple average of the scores given by the evaluators, but to discuss and to arrive at a collective view. In the case of proposals where a panel member declared an interest, that member left the room for the discussion and also the ranking. The rapporteurs synthesised the views and suggested collective scores. One evaluator who had made a primary assessment composed the text of the report, which was adopted as the common view of the subgroup and panel.

Despite the diversity of the proposals in terms of research area and also the diversity of the panel, the discussions were primarily concerned with the scientific quality (including quality of scientific output) and the budgets of the proposals. The issue of whether a facility was really "world class" was often raised, as was the size and importance of the potential community of users. The primary assessments were not of course in complete agreement, but they were mostly reasonably close. In a few cases, the evaluators had consulted with other experts, as advised in the briefing meeting, to obtain opinions about scientific areas not within their direct competence.

The level of detail in the discussion demonstrated that the evaluators had read the proposals carefully and considered opinions were given rather than quick impressions. Several mentions were made of looking at web sites of facilities. In fact, rejected proposals often took 20 minutes or more to discuss. In some cases no changes were made to budgets (usually where top marks had been awarded), and in others careful consideration and matching to the rules led to recommended cuts. Budget information was not always clear and there were striking differences according to country which the evaluators found hard to interpret sometimes. There was no question of favouring proposals from particular parts of the EU or associated states over considerations of scientific quality.

There was discussion in one subgroup about the precise nature of a research infrastructure and what a network contract should seek to achieve. Given the openness of the programme to all areas of research, this is not surprising. The Commission staff clarified the definitions effectively.

The writing of the common comments was done extremely carefully, with general guidance from the Commission staff that it should give a very clear message, particularly where cuts in budget were sought or where proposals were below the quality threshold. The process was completed efficiently, although it did require a lot of work from the evaluators.

It is not a surprising observation that the subgroups did work in slightly different ways, and that the rapporteurs influenced how the subgroups worked. Mitigating against this was the high degree of cross membership of the groups and the presence of Commission staff in each group. In all groups, ranking of the proposals was rapid and passed without disagreement.

10. The Panel Meeting - Ranking of the Proposals

On the third and final day, the Panel considered the ranking of all the proposals. Some experts were concerned about the comparability of scoring between the subgroups. The median scores for successful proposals varied from 74 to 92, and success rates varied too. The panel as a whole rejected any "correction" as arbitrary. The panel as a whole did not want to fund all successful proposals by cutting budgets or access months across the board, or by cutting down all RTD projects. Such solutions were rejected as being unfair to highly scoring proposals. The Chair and Vice chair were important in structuring these plenary discussions.

The ranking then took place efficiently. The commission rules on giving priority to less favoured regions for equally weighted proposals were followed. The overall ranking followed the ranking order indicated within the subgroups and where these were equal, proposals with a higher score in categories related to scientific quality were ranked first.

11. Reflections on the Evaluation

The Chair organised a discussion among the Panel reflecting upon their work.

Experts who had served in the first call all agreed that the discussion of each proposal in the subgroups was a great improvement. The debate between specialists (members of the subgroup) and non-specialists (rapporteurs) was seen as being a rigorous way to make the judgements. The independent monitor found that the discussions within the groups were detailed and allowed a very fair treatment of each proposal.

The precise role of the rapporteurs was not set out and this was suggested for the future, to promote equivalence between subgroups. In the view of the independent monitor, the specified role should include ensuring that the comments and the scores given are consistent, probing the experts about their judgements and involving the whole subgroup in the discussions.

Some experts favoured giving more weighting to scientific quality criteria, since these determined the final ranking. This should be considered by the Commission. Some evaluators felt that more information about the infrastructures could be provided with the proposals (ie mid term review results). Some evaluators did not wish to be anonymous but to be able to enter into a dialogue with the proposers (eg when aspects of the proposal were unclear). This might prove difficult unless done in a consistent way, probably mediated by Commission staff.

In the view of the independent monitor, the rules on conflict of interest for evaluators are appropriate, except in the case of nationality. In a basic science programme, national interests are not relevant for expert evaluators (although this might not be the case for policy makers). There was no evidence of any national prejudice in discussions about proposals.