CORDIS Archive

Monitoring, Evaluation & Assessment Activities

Overview of the EU RTD Programme Monitoring and Evaluation System

In connection with the Fourth Framework Programme and with the aim of fully responding to legislative requests (cf. Annex for the current main legal basis) - while building on “best practice” - the Commission Services set out the new evaluation scheme in 1994 drawing on experience from the extensive previous work.

The scheme includes a continuous monitoring reporting annually, and a five-year assessment carried out mid-way through programme implementation so that it includes two subsequent programmes and produces results in time for the presentation by the Commission of the next Framework Programme proposal.

As a consequence, the five-year assessment combines an ex-post evaluation of the previous programme, a mid-term appraisal of the on-going one, and recommendations for future activities. The monitoring is carried out by the Commission services with the assistance of external experts, the five-year assessment is conducted by such experts.

The monitoring as well as the assessment are carried out for all Specific Programmes in parallel so that input is produced at the Framework Programme level in a synchronised manner.

This new scheme was formally introduced in 1995 further to the positive opinion of CREST (a body composed of Member States representatives advising the Commission and the Council on S&T; related matters).

Continuous Monitoring versus 5-Year Assessment

The monitoring and the 5-year assessment are different exercises. The assistance of external experts in the monitoring provides a quick response mechanism to programme development. It is a light exercise on an annual basis, designed to provide a constructive independent critique on key issues. In contrast, the five-year assessment is a structured multi-annual evaluation conducted by the external experts with approximately four year intervals, thus it is a broader assessment of objectives, performance, exploitation, etc. The two exercises are different and must remain distinct from each other.

1) Continuous monitoring

For the Specific Programmes, this exercise aims at ensuring the cost effective implementation, examining progress in relation with the original objectives and whether these objectives, priorities and financial resources are still appropriate to changing circumstances. For the Framework Programme, the aim is to monitor overall progress as regards the major objectives, and to examine whether the objectives, priorities and financial resources are still appropriate in the overall context.

The day-to-day monitoring of the Specific Programmes’ implementation is carried out by the programme management within the Commission Services. In addition, panels of external experts provide an independent view once a year (the process itself lasts over a few months) on the progress of implementation. The panels, one for each of the Specific Programmes (and one for the Framework Programme) give advice on key issues relating to programme development and thus help programme management and programme committees to identify and correct weaknesses. In the light of the results of the monitoring, the Commission may submit proposals to adapt or supplement the Specific Programmes or the Framework Programme.

The continuous monitoring exercises aim also at collecting data which are useful for the five-years assessments. Moreover, the sets of the annual monitoring reports provide, notably, the five-years assessment panels with information on the effectiveness of programme implementation.

Both at Specific Programme and Framework Programme levels, the following issues are required to be addressed by the panels:

• efficiency and transparency of the programme management (including calls for proposals, information to applicants, the assessment and selection process, contract negotiation and disbursement of funds), and internal Commission co-ordination;

• consistency of the selection of projects with the initial objectives and the work programme, and the extent to which selected projects or clusters of projects fulfil the wider policy objectives of the European Union (in particular in areas of relevance to the programme concerned);

• use of specific measures and support activities (e.g. to support SMEs, improve dissemination, etc.), and participation in the programme of firms and institutions from less favoured regions;

• appropriate follow-up of previous evaluation/monitoring recommendations;

• the progress and output of projects against the original targets set; and

• aspects of flexibility to respond to the needs of society in the light

• of changing circumstances.

The panels are also invited to produce recommendations for the future indicators to be used for monitoring as well as for the monitoring process itself.
The Framework Programme level exercise is mainly a synthesis of the Specific Programmes' monitoring (including core indicators - see below), summarising progress and giving emphasis to the main issues which have emerged from the analysis. Nevertheless, the panel’s report is more than simply the sum of the specific programme monitoring reports. Therefore, the following additional issues are considered:

• cases where the independent monitoring experts consider the results will have a significant impact, or where poor performance requires further examination;

• as appropriate, consideration of Community RTD objectives as well as synergies between Specific Programmes; and

• changes that may be needed to the balance of the Programmes or to the strategy for implementation, in the light of experience and changes in the wider environment.

2) Five-year assessment

The overall objective of this exercise is to provide input to policy formulation and decision-making, based on feedback from implementation. For the Specific Programmes, the five-year assessment aims at evaluating the activities carried out within the fields covered by the programmes and their management during the five years preceding the assessment. In particular, the following are examined: the relevance of the initial objectives considering major new developments, the cost-effectiveness of programme implementation, and effectiveness in achieving the original objectives. The exercise also results in identification of major achievements and lessons learned from programme implementation, and provides recommendations for future activities. It is expected that these evaluations can be carried out efficiently using the data collected during the monitoring operations and the monitoring reports themselves.

The Framework Programme five-year assessment, which is based on the Specific Programme assessments and combines them at a higher level, goes beyond the evaluation of past and current activities and considers the next Framework Programme taking into account also the working documents of the Commission available at that time. Consequently, this five-year assessment combines an ex-post evaluation of the previous Programme, a mid-term evaluation of the on-going one and an ex-ante appraisal of future activities.

Both at Specific Programme and Framework Programme levels, the following key issues are required to be addressed by the panels:

• relevance, i.e. whether the initial objectives are still valid against new S&T; developments and socio-economic conditions;

• efficiency, i.e. whether the objectives have been pursued in a cost effective manner; and

• effectiveness, i.e. whether the initial objectives have been achieved, or, for longer term strategy and objectives, if progress is sufficient. Moreover, whether the “European added-value” has been adhered to and the results have been disseminated/exploited.

At the Framework Programme level, in addition, the panel is requested to pay attention to the coherence between the Community and national S&T; policies with a view to enhancing their mutual consistency, and to aspects of co-ordination with other international S&T; policies or programmes.

Annex

DECISION No. 182/1999/EC (22 December 1998) OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL CONCERNING TO THE FIFTH FRAMEWORK PROGRAMME OF THE EUROPEAN COMMUNITY FOR RESEARCH, TECHNOLOGICAL DEVELOPMENT AND DEMONSTRATION ACTIVITIES
(1998 TO 2002)

ARTICLE 5

1. The Commission shall continually and systematically monitor each year, with the help of independent qualified experts, the implementation of the Fifth Framework Programme and its specific programmes in the light of the criteria set out in Annex I and the scientific and technological objectives set out in Annex II. It shall assess, in particular, whether the objectives, priorities and financial resources are still appropriate to the changing situation. Where appropriate, it shall submit proposals to adapt or supplement the framework programme and/or the specific programmes, taking account of the results of this assessment.

2. Before submitting its proposal for a Sixth Framework Programme, the Commission shall have an external assessment conducted by independent highly qualified experts into the implementation and achievements of Community activities carried out during the five years preceding that assessment in the light of the criteria set out in Annex I , the scientific and technological objectives set out in Annex II and the implementation of this Decision via the specific programmes based thereon. The Commission shall communicate the conclusions thereof, accompanied by its comments, to the European Parliament, the Council, the Economic and Social Committee and the Committee of the Regions.

3. The independent qualified experts referred to in paragraphs 1 and 2 shall be drawn in particular from scientific, industrial and user communities and chosen on the grounds of their expertise and personal aptitude by the Commission, which shall take account, in a balanced fashion, of the various research players.
   The Commission shall make known the full list of experts and their individual qualifications following their appointment.

4. The Commission shall regularly inform the European Parliament and the Council on the overall progress of the implementation of the Framework Programme and the Specific Programmes.

CRITERIA FOR SELECTING THE THEMES AND OBJECTIVES OF COMMUNITY RTD ACTIVITIES

1. The European Community's RTD policy is directed towards strengthening the scientific and technological bases of Community industry and encouraging it to become more competitive at international level, while promoting all the research activities deemed necessary by virtue of other Chapters of the Treaty. It shall also contribute to promoting the quality of life of the Community's citizens and to the sustainable development of the Community as a whole, including the ecological aspects. Its implementation is based on the twin principles of scientific and technological excellence and relevance to the abovementioned objectives.
   Moreover, in pursuit of a cost-benefit approach dictated by concern for optimum allocation of European public funding and in accordance with the subsidiarity principle, themes for the Fifth Framework Programme and the related objectives are selected on the basis that the Community shall take action only if and insofar as the objectives cannot be sufficiently achieved by the Member States.

2. In application of the foregoing principles, the framework programme shall be defined on the basis of a set of common criteria, divided into three categories:

   • Criteria related to the Community "value added" and the subsidiarity principle:

     • need to establish a "critical mass" in human and financial terms, in particular through the combination of the complementary expertise and resources available in the various Member States,
     • significant contribution to the implementation of one or more Community policies,
     • addressing of problems arising at Community level, or questions relating to aspects of standardization, or questions connected with the development of the European area,
     • 
       so as to select only objectives which are more efficiently pursued at the Community level by means of research activities conducted at that level.

3. Criteria related to social objectives:

   • improving the employment situation,
   • promoting the quality of life and health,
   • preserving the environment,
   • 
     in order to further major social objectives of the Community reflecting the expectations and concerns of its citizens.

4. Criteria related to economic development and scientific and technological prospects:

   • areas which are expanding and create good growth prospects,
   • areas in which Community businesses can and must become more competitive,
   • areas in which prospects of significant scientific and technological progress are opening up, offering possibilities for dissemination and exploitation of results in the medium or long term,
   • 
     in order to contribute to the harmonious and sustainable development of the Community as a whole.
   
   

5. The criteria referred to in paragraph 2 will be used, and where necessary supplemented, for the purposes of the implementation of the Fifth Framework Programme, in order to define the specific programmes and select research and technological development activities, including demonstration activities. The three categories of criteria will apply simultaneously and must all be met, although to a different extent from case to case.

Monitoring, Evaluation & Assessment