European Commission logo
polski polski
CORDIS - Wyniki badań wspieranych przez UE
CORDIS

DEvelopment of MEthods and Tools for Evaluation of Research

Periodic Report Summary - DEMETER (Development of methods and tools for evaluation of research)

Project context and objectives:

The Lisbon agenda to increase competitiveness of European countries by the knowledge has focussed attention on research and technological development (RTD) and human capital policies that are supposed not only to increase competitiveness, but also to boost growth and employment, to reduce poverty and inequalities and to take up the challenges of demography and of environment.

We are now in a new situation with 'great expectations of science and technique (ST) as regards tackling the multitude of challenges'. Echoing to this, a lot of reports are pointing out the weaknesses of Europe ST system: the rather low RTD intensity compared to the United States of America (USA) one, the lack of researchers, the insufficient attractiveness of European countries for researchers and the lack of coordination for Europe as a whole… Nowadays, many people are feeling more or less confusedly the need for strong research policies at both national and European levels. That is the reason why policy makers, researchers, private business would wish to have a better quantification of the results of RTD efforts and, for policy makers and stakeholders, a good evaluation of foreseen or already implemented RTD policies.

But the evaluation of such policies is, in the 'state of the art', very uneasy:

1. at the firm or sectoral level, RTD performance is difficult to catch and to quantify; the impact of public incentives on RTD decision is also not very clear. It is the case of subsidies, with great uncertainties onto leverage effect of public funding and the effects of other incentives are also uneasy to quantify;
2. at a more global or macro-economic stage the assessment is even more difficult, the interdependencies and the composition effects darkening the results.

How the theory and practice of economics on research and technical change can help us for this evaluation?

In the representation that separates the process of RTD and inventions on one side and the process of adoption and diffusion of innovations on the other, the concepts of 'externalities' and 'market failures' play a major role and justify the research policies. These externalities are numerous and linked to the status of innovation that is not strictly appropriable and to the transfers of innovation rents. The knowledge spillovers are in the core of the new developments (endogenous growth and other streams). They justify the non-decreasing returns hypothesis at macro level and above all, the needs for research policies because the social returns of RTD are more important than the individual ones.

But the productivity of RTD is today in debate from theoretical and conceptual points of view.

At first there are the links between RTD and human capital (Vandenbussche, Aghion and Meghir, 2006) that are renewed from a theoretical point of view with the 'technological frontier': the productivity of RTD will depend on the distance to that frontier, while the contribution of high skilled labour to innovation increases with the proximity of firms, sectors or countries to it. This feature is particularly important for Europe that aims to come closer to the frontier.

A second important point with which policy makers are concerned is the observed lag between RTD investments and the increase in factor productivity ('productivity paradox'). If this lag is important, RTD expenditures that are costly will not be rewarded during a long time and then the RTD policies will be threatened by deficits and inflationary pressures in the short term. Do theories' advancements help us on that subject?

From a conceptual point of view, there are a lot of advancements that modify the perception we have on RTD performance. The spillovers effects between firms and nations are today so important that the links between the localisation of RTD activities and the localisation of innovations are more tenuous.

The growing importance of externalities, based on knowledge transfers resulting from RTD expenditures of firms and public laboratories and from the rise of innovation capacity, coming through the development of 'intangible capital', must also enter into the conceptual framework. In that new perspective, RTD is only one input of innovation among many others, such as information and communications technology (ICT), organisational capital and human capital… all variables that modify the ability to exploit the knowledge stock (David and Foray [1995]).

From an empirical point of view, important works grounded on improved econometrics methods (Mairesse and Sassenou, [1991]), tried to assess for the performance of RTD at sectoral or individual firms level, using RTD stock resulting from the accumulation of research expenditures flows. These works where extended to include in the knowledge variable of firms and sectors the spillovers arising from RTD expenditures from other sectors and other countries, with the use of technological matrices (Johnson [2002], Verspagen [1997]).

Technology and knowledge externalities have been subject to a lot of applied works and the recent developments of statistics and information allowed by the new computers capacities has entirely renewed the perception of the former chain linking RTD to innovation. The science, technology and innovation indicators (STI) play a major role and their use must be widened in order to take into account the emergence of new sorts of knowledge (Freeman and Soete [2007]). This may widen the design and the scope of research and innovation policies, involving for instance the localisation of RTD activities, accompanying measures that reinforce the RTD efficiency and enhance simultaneously the innovation capital constitutive of the European innovation capacity (ICT, knowledge, human and organisational capitals).

But the whole assessment of RTD policies, at national or European level, must be realised with applied macro-economic modelling, if one wants to produce all detailed results on competitiveness and growth, employment and other sustainable development indicators.

Two types of models are generally used: econometric models and general equilibrium ones. The first are based on econometric works that ground modelling in temporal data and with some precautions can be used to build short medium term forecasts and more long term business as usual scenarios. They have been considered for a long time as models with 'neo-Keynesian flavour' that is to say mainly commanded by demand, but this feature can be considered today as over passed in some models that built in supply side properties and rational expectations.

The general equilibrium modelling is more grounded in main stream micro-economic theory: the optimisation of behaviours and equilibrium on all the markets. They are generally 'calibrated' and this prevent them to be used for forecast but they can give long term assessment in a rigorous framework and compute the utilities or surplus variations that are induced by the implementation of policies. These instruments are for these reasons complementary and very often used simultaneously for policy assessment.

What is the 'state of the art' of that modelling for RTD assessment objectives? During a very long period these models adopted an exogenous technical change approach even if in some modelling, technical change was embodied to capital.

It's rather recently that a genuine endogeneisation of technical change on RTD was formalised by Coe and Helpman [1995] on a macro model adopted for trade policies. But this approach was macroeconomic and it is even more recently that was incorporated in the modelling the new dimension of knowledge externalities in the endogeneisation of technical change, at a detailed sectoral level (see Fougeyrollas, Le Mouël and Zagamé [2001] [2005]).

The overall objective of the DEMETER project was to build a system of tools based on applied modelling that can be used for the ex-ante evaluation of research and innovation policies at sectoral and European level.

For this purpose, DEMETER used two already existent detailed large scale models: Nemesis and GEM-E3. Both models already included endogenous technical change mechanisms and were already used to assess for RTD and innovations policies. The models are complementary, in the sense that one, Nemesis, is econometric and well-suited to analyse the short to medium term dynamics, while the second GEM-E3, is useful for analysing the long term impacts of policies.

Starting from this existing experience accumulated by Nemesis and GEM-E3 consortiums, the aim of DEMETER was then threefold and organised in three parts:

1. the first part performs new theoretical and empirical breakthroughs on key challenges for European innovation strategy, mainly on the role played by knowledge spillovers, the relation between ICTs and productivity, the complementarities/substituabilities between tangible and intangible assets and the productivity breakdown in services;
2. the second is the reshaping of the macro-economic models Nemesis and GEM-E3, based on the results reached in the first part on the modelling of knowledge externalities, on the links between innovation policy and economic performance and the role played by ICTs and intangible capital;
3. the last part consisted in the assessment of various European Union (EU) and national RTD and innovation policies with the macro-economic models all along the project period, focusing on EU Framework Programmes (FPs) and RTD and innovation aspects of Europe 2020 strategy.

Project results:

Some theoretical and empirical considerations

Strategies and challenges

This part includes different subjects. The first one is on the notion of innovation strategies by firms: innovation strategies comprise the total of decisions on the type and quantities of resources to use, for innovation activities in firm, the goals of these activities and the way the innovations are protected. A second research tries to link this notion of innovation strategies to the relationship between employment and innovation: do different innovation strategies correspond to different impacts of innovation on employment? To investigate this question we used micro econometric analysis on a smaller set of countries. Another research is defining the main challenges ahead for Europe and try to assess for, in this context, the costs of a non innovative Europe. The problem of productivity, especially in services when compared to US one is raised up. One important point for the explanation of this 'European disease' is linked to the diffusion of ICT.

The ICT question

In order to deepen the question, an important effort was made on the analysis of ICT in the aim of implementing the results in modelling. ICT are at first viewed as general purpose technologies (GPT); GPT offers an attractive conceptual tool to bridge the analysis of the micro economic structures of technological change. Several issues are tackled after a descriptive analysis of ICT data: the complementarities between ICT and capital, their substitutability with intermediate inputs, the externalities linked to ICT, etc. At the end, based on a new type of growth accounting, a research tries to measure the contribution of ICT to growth.

The knowledge spillovers

If the knowledge spillovers play a major role with the GPTs and then with ICT, more generally, all the productive and innovative activities are concerned by these externalities. A reliable assessment for research and development (R&D) policies need an exact determination of these flows of knowledge that affect the R&D productivity. If a lot of work has been already achieved on intersectoral spillovers, on the other hand the international spillovers are less precisely assessed and a simultaneous precise assessment for intersectoral and international spillovers has to be done.

Research on modelling

Research on modelling is partly inspired by the results of part one; it is the case for the ICT and for knowledge spillovers; but other improvements of models have been implemented.

New data bases on R&D expenditures

R&D expenditures data are provided by the Organisation for Economic Cooperation and Development (OECD) in the International Standard Industrial Classification (ISIC) revision three nomenclature and they were harmonised in order to fit the sectoral disaggregation of Nemesis and GEM-E3.

ICT in Nemesis

All the results on ICT were used in order to estimate new production functions with ICT capital.

Knowledge spillovers

The knowledge spillovers matrices built by the United Nations University (UNU) Maastricht Economic and Social Research Institute on Innovation and Technology (MERIT) part one were implemented in the Nemesis and GEM-E3 models, then analytical shocks on R&D allocated with different criteria have been implemented in order to make a sensitivity analysis of the new models.

Extension of GEM-E3 and mechanisms modification

On the GEM-E3 model, three tasks have been achieved: the extension of the model so as to include all major countries regarding R&D activities, the improvement of the R&D supply mechanisms of the model and the respecification of firms' demand function for R&D.

Policy assessment

The models were used for R&D policy assessment. The novelty, when compared to the former assessments, is the occurrence of crisis. A first issue of the research is on 'the role of R&D policy during crisis and beyond'. Would the crisis reinforce the usefulness of research support policies? Is it time to reattempt the Barcelona objective of 3 % RTD effort? What can be the role of FP in that context?

But the Nemesis model was also used in order to assess for different European and national policies: policies already decided like the FP or Common Strategic Framework (CSF) and European Research Area (ERA); policies not already decided, in order to sustain reflection on new possibilities for increasing research: redeployment of structural funds, public procurements for innovative goods and services, etc.

Potential impact:

The DEMETER project allowed improving existing applied modelling tools for providing precise ex-ante impact assessment of RTD and innovation policies both at national, European or world level (GEM-E3).

Before DEMETER, the two models were already widely used for assessing structural policies. Nemesis was used notably to assess for the 3 % RTD effort Barcelona objective for Europe (Brécard D., Fougeyrollas A., Lemiale L., Le Mouël P, Zagamé P., 2006 and the impacts of the European Framework Programmes for research and GEM-E3 was extensively used to evaluate the EU 'Climate and Energy' strategy (Saveyn, B., Van Regemorter, D. and Ciscar, JC., 2011).

During the DEMETER project, Nemesis was continuously used to assess for EU policies for Research and Innovation. Detailed presentations of these policy assessment can be find of the DEMETER website (see http://www.demeter-project.eu online) and on the European Commission portal (see http://ec.europa.eu/research/social-sciences/ online). These assessments concern notably:

1. the role of R&D policy in the new context of economic crisis ('The costs of a non-innovative Europe: What can we learn and what can we expect from the simulation works', Paul Zagamé - 27 September 2010; 'The costs of a non-innovative Europe: Challenges ahead', Luc Soete - 21 September 2010);
2. the impacts of re-attempting the Barcelona objective, of doubling the Eighth FP budget together with a EU's financial reform, to tackle the crisis ('R&D effort during the crisis and beyond: Some insights provided by the Nemesis model simulations', Arnaud Fougeyrollas, Pierre Haddad, Boris Le Hir, Pierre Le Mouël and Paul Zagamé - 20 May 2010);
3. the consequences of the FP7 2011, 2012 and 2013 call on European economy and employment (Arnaud Fougeyrollas, Pierre Le Mouël and Paul Zagamé 24 September 2010, 14 July 2011 and 31 May 2012);
4. but the Nemesis model was also used in order to assess for different European policies: policies already decided like the FP and CSF, or ERA policies not already decided, in order to sustain reflexion on new possibilities for increasing research, such as redeploying of structural funds, public procurement for innovative goods and services, etc.

The improvements of the models realised during the DEMETER project will allow in the future to pursue RTD and innovation ex-ante policy assessments with new focuses on the specific impacts of ICT and intangible assets (R&D but also non scientific R&D, e.g. mineral exploration, copyrights and licence costs, new product development in final industries and new architectural and financial designs; economic competencies; brand equity: advertising expenditures and market research; and firms specific resources: human capital, purchased and own account development of organisational structures), with the model development realised with Nemesis and on international aspects of R&D policies, with the introduction of a R&D module in the world version of GEM-E3 model. Also, the new dataset and conceptual framework for modelling knowledge flows and their effects on the economic performance of the European economy carried out by University of Maastricht (UM) and UNU-MERIT (Huub Meijers and Bart Verspagen), will allow, by better capturing the knowledge flows between sectors and between countries, improving importantly the assessment of R&D policies with the two models.

For the dissemination of the results, a project website was established by the coordinator of the project (see http://www.demeter-project.eu online), which is partly available to the public. Each partner will continue to contribute to this website by writing articles, which will be submitted to academic peer-reviewed specialised journals. The coordinator, Société Européenne d’Economie (SEURECO) Équipe de Recherche en Analyse des Systèmes et Modélisation Economique (ERASME) is seeking the opportunity to publicise the work through special issues of such journals or a book giving the opportunity to all partners to contribute individually as well as allowing for an overall appreciation and synthesis of the results of the project. The final publishable report of the project will serve as a basis for these academic dissemination objectives.

Some projects results were also put available to a large audience through press released, notably by Máire Geoghegan-Quinn, the EU Commissioner for Research (see http://www.eubusiness.com/topics/research/fp7-calls-11/ and http://presseeuropa.fr/press-releases/8-1-milliards-d-euros-d-investissements-dans-la-recherche-et-l-innovation-pour-creer-de-la-croissance-et-des-emplois online).

Project website: http://www.demeter-project.eu