Skip to main content

Impact of Research on EU Agriculture

Final Report Summary - IMPRESA (Impact of Research on EU Agriculture)

Executive Summary:
The IMPRESA project has focused on assessing the socio-economic impacts of agricultural research in Europe. Its chief aims have been to measure, assess, and comprehend, on aggregate and in depth, the output, outcomes and impacts of agricultural research, using both qualitative and quantitative methods. Its overall intention has been to maximise economic, environmental and social benefits derived from diverse strands of public, private and not-for-profit research investment expenditure.

To achieve this, it documented existing knowledge of European agricultural research investment, using alternative sources to fill gaps in official series of expenditure statistics, and complemented numerical data with more detailed assessment of institutional trends and prospects for future activity. It has examined a diverse range of case studies, to understand the process by which science-based innovation generates ultimate impacts, exploring factors that hindered or enhanced these impacts. It developed a series of quantitative analytical techniques that, at various scales and using different approaches, developing a broader account of overall impacts from agricultural research investment.

Insights into important contemporary changes in the European agricultural knowledge and innovation system were gathered from a survey of 20 European countries. This provided information on data availability and investment trends at national and European level. Five countries accounted for more than 70% of planned public agricultural research; in the EU28, the aggregate declined by 7% in nominal terms between 2008 and 2013. National research priorities are increasingly aligned with European Framework Programmes. Improved monitoring of agricultural research expenditures in the short term should rely on an annual report surveying investment by responsible ministries; in the medium term on an annual survey of public research organisations; and ultimately by obliging countries to produce more detailed agricultural research statistics.

Lessons from case studies of science-based innovation are based on Participatory Impact Pathway Analysis. This showed that expected impacts were at least partially met at both farm and territory levels; unintended direct impacts and unexpected indirect impacts occurred; and enabling and hindering factors were not always directly linked to research. More ex-post impact evaluation is required to enrol researchers into a new ‘culture of impact’; also to plan early for impact and involve key stakeholders (including private companies) at an early stage in the research. Much scope exists for strengthened research and innovation policy, through coordinated integration of research and innovation support instruments and agricultural policy, and better evaluation through improved availability and access to research data for impact assessment.

Econometric results show that public research expenditure positively impacts on productivity. Returns are in the range 7-15%, after a lag extending 9-18 years. Social and environmental impacts are positive, but depend greatly on intermediate factors and with complementary effects from public or private research. Sector-level decomposition indicates faster agri-food technical change is than the overall average, with stronger transmission of change from the food industry to agriculture than vice-versa. Private food-processing research investment has positive impact on performance, with differences across sub-sectors and regions, with greater impact for intermediate processing than for prepared food, and more responsiveness in Canada and Eastern Europe than in the USA, Japan or Western Europe. Exploring the pathway from research to performance at the farm level indicates the importance of innovation, boosted on farms by networking connections with knowledge brokers and value chain actors. Links to research awareness have more effects on output quality, less on production levels and costs. Overall, the positive impacts of research need qualifying. There are difficulties with causality, describing intermediate proxies in the impact process and the effect of spillovers from other fields of research. More scope for mixed research methods should be exploited.

Project Context and Objectives:
Improving agricultural research impact is an important goal for the EU. IMPRESA has studied the process of research impact across Europe, using a survey of levels and trends of research expenditures by public and private sectors, case studies identifying impact pathways of individual science-based innovations, and quantitative analyses of the relation between research investments and their final impacts.

The European Union’s long term growth policy gives a prominent role to research-based innovation. Initially, the Lisbon Strategy aimed for a competitive and dynamic knowledge-based economy. Its successor, the Europe 2020 Strategy, has targeted smart, sustainable and inclusive growth. Understanding the agricultural science base, from which innovation in agriculture should predominantly arise, is an important first step in enhancing innovation and beneficial impacts within the sector. A substantial amount of activity is devoted understanding the impacts of science on innovation and the benefits to wider society in low-income countries (particularly in relation to the Millennium Development Goals: see, for example, CGIAR Science Council, 2005). It is therefore of some concern that relatively little is known overall about the impact of agricultural science in Europe. In the ex-ante impact assessment of the reformed CAP introduced in 2014, the annex on research and innovation noted that “it is not possible to draw a complete picture of the overall (agricultural research) effort since there are no data on private investments” (EC, 2011: 5).

The Global Food System is being affected, simultaneously, by three major factors. First, population numbers are rising, but moreover are magnified by changing dietary preferences which favour animal proteins. Second, in order to reduce greenhouse gas emissions, energy usage is being decarbonised, and this is making substantial alternative demands on arable land for non-food outputs, especially biofuels. Lastly, and in the longer term, processes of climatic and environmental change will have uncertain but varied impacts on productivity.

In Europe, as in many other OECD countries, public support for agriculture is at the same time under pressure from budgetary austerity, following the financial crisis. The combined effects of these will reinforce pressures on food price levels and increase the likelihood of future price volatility; in turn, these trends raise concerns about food security, poverty and social welfare, and even constrained prospects for future economic growth.

The overall solution to this triple crisis is to improve productivity while at the same time reducing negative environmental impacts. Achieving this desired transformation of the agri-food complex will require public policy action on a number of fronts, including the review and reform of trade policies, regulation, and agri-environment incentives, alongside consideration of the food processing and distribution systems which link primary producers with final consumers. A crucial contribution can be made by scientific research. It can improve the effectiveness of policy reforms; help meet the objective of internalising external environmental costs; promote more even spatial development; and support progress towards more stable and fairer consumer prices. IMPRESA will explore the impact and importance of scientific research, in terms of achievement of these broad policy objectives, and demonstrate how public resources devoted to research can best be prioritised and focused. This venture is of considerable importance, for two main reasons.

Firstly, the small average size and large number of farm businesses precludes their individual engagement in formalised research and development activities, and the types of innovation that it can engender. This can be partly offset by research activities of other sectors, which supply inputs to agriculture and process its outputs. However, poorly defined property rights and the relative site- specificity of many types of innovation make it difficult for farming, as a sector, to initiate, control and adopt research which fully addresses its needs.

Secondly, evidence suggests that SRA is a highly cost-effective means of improved productivity, and it has been critical in identifying, addressing and mitigating adverse environmental consequences of traditional intensification. Because of the magnitude of the challenges being faced, it has become increasingly important that the limited funds devoted to research, whether from public (national government or European Framework Programmes) or private (commercial or not-for-profit) sources, are used in the most effective possible manner.

The aims of the IMPRESA were two-fold. The first was to describe the contemporary evolution of public and private agricultural research (bearing in mind that recent scientific and supply chain developments blur the boundaries of the discipline, as traditionally defined). The second was to explore its resulting impacts, using a variety of qualitative and quantitative approaches. IMPRESA’s approaches address specific issues for agricultural science and its translation into innovative technological solutions and ultimate impacts to address structural problems of low agricultural productivity.

IMPRESA achieved triangulation, validation and utilisation of these two sets of results in line with the recommendations of the most recent report of the European Standing Committee on Agricultural Research (EU SCAR, 2012), which proposes additional emphasis on innovation-driven research to complement current success in science-driven research. IMPRESA extensively embraced the active engagement of end-users in its analysis and reporting, including stakeholder networks centrally in the process of researching impact itself, subject to safeguards to avoid conflicts of interest. This engagement has consolidated and enhanced the effectiveness of dissemination activities, and the result has been more practical and relevant recommendations for European agricultural science policy evolution and effectiveness.

The aims of measuring, assessing and comprehending the impact of all forms of European SRA on key agricultural policy goals, including farm level productivity but also environmental enhancement and the efficiency of agri-food supply chains, have been achieved through five major objectives:

1. Exploration of the scope and relevance of scientific research activity for the objectives and implementation of agricultural policy, involving participation of expert informants, that will provide guidance for improved accuracy in assessing the volume, type and current trends in activity; in turn, this should allow more focused effort on appropriate empirical strategies and development of protocols for future monitoring
2. Assessment of the agricultural science impact base, across Europe, in terms of the efforts of national and supranational public authorities, commercial and not-for-profit enterprises through expert-led surveys of activity in the EU-28, EEA-4 and Switzerland
3. Implementation of a comparative case-study investigation of contrasting implementations of science-based agricultural innovation (with regard to productivity, environmental impact and rural development consequences) that will identify both common and specific mechanisms involved in research, dissemination, adoption and impact
4. Testing of quantitative approaches to research evaluation, in countries selected on the basis of complementarity with in-depth case-study inquiry, to determine rates of return on SRA, the distribution of benefits across the agri-food supply chain, and the impacts of such research on performance of enterprises allied to the farming sector
5. Identification and dissemination of key characteristics of the evaluation of SRA to the main stakeholders (policymakers, extension services, agri-food input and processing sectors, land use and environmental non-governmental organisations) through a participatory framework that will promote a more flexible, responsive and effective alignment between the research community and major social objectives concerning food security, environmental quality, energy efficiency and resilience of farming systems.

Project Results:
Learning from the Past

In early 2014, FAO as a partner in the IMPRESA team moderated an email conference on ‘Approaches and methodologies in ex-post impact assessment of agricultural research: Experiences, lessons learned and perspectives’ to develop the methodological foundations for IMPRESA’s examination of the impacts of agricultural research in Europe.

618 people subscribed to the conference and 59 submitted at least one message. Of the 109 messages posted, around 30% each came from people living in Africa and Europe; 15% from Asia; 13% from Latin America and the Caribbean; 7% from North America and 6% from Oceania. Messages came from people living in 38 different countries, and 58% were posted by people living in developing countries. A total of 25 messages were from people working in universities, 23% from those working in national research centres and 22% from independent consultants. Fourteen per cent came from people working in the international agricultural research system, 7% from people in inter-governmental organisations and 5% each from non-governmental organisations (NGOs) and government ministries and bodies.

The issues raised by the participants over the 4 week conference period mostly addressed micro-level aspects of ex-post impact assessment of agricultural research (henceforth epIA-AR), typically looking at the impacts of specific research projects in one part of a country, rather than macro-level aspects, which focus on the impacts of investments in agricultural research (or in one of its sectors) at the national level.

One of the main topics of discussion in the conference was participation. Contributors felt that there was a strong need to ensure participation of key stakeholders and beneficiaries in epIA-AR processes, exploiting appropriate communication tools to ensure the full participation of some stakeholders, particularly small farmers. Participation in the epIA-AR processes was perceived as an inherently positive feature and also has practical advantages in terms of improving access to useful data and making it easier to communicate the findings of impact assessments with key stakeholders and beneficiaries.

Quantitative and qualitative methodologies are used in epIA-AR; the former typically involves data recording and statistical testing, whereas the latter usually involves interviews with key individuals, or group discussions. The conference raised differences in opinion about the relative merits of these methodologies, although there was a general agreement about the mutual benefits of using both. Difficulties in sourcing the data needed for quantitative methods, particularly in less-developed countries, were also discussed.

Several participants highlighted the difficulties of attributing changes that occur within complex and dynamic agricultural systems to specific interventions derived from research. They noted that the impacts attributed to a specific research project may partially be the consequence of other interventions in the same target area that were not considered in the analysis. Some saw this as a weakness of attribution analysis and argued that contribution analysis was a better alternative (see Forss et al. 2011, for example).

The importance of getting relevant and reliable data for epIA-AR was underlined by many conference participants. One key theme to emerge was the need for ensuring that appropriate data are collected and made available for future evaluation studies. Two particular practical challenges were noted: acquiring the data required to build indicators of the social and environmental impacts of agricultural research and getting reliable data on adoption rates. The later can be problematic as farmers often modify the innovations derived from research. Relevant baseline data also needs to be collected in order to assess the impacts of the activities of research institutions.

While the main focus of the e-mail conference was on how to do epIA-AR, there was also ample discussion on why and when it should be done and which impacts should be assessed.

In relation to why, participants identified two main objectives: accountability and learning. Conference participants pointed out the link between objectives of any epIA-AR and the type of methodology (quantitative and/or qualitative) used, and/or the type of research initiatives assessed.

In relation to when, the consensus among participants was that impact refers to the long-term effects of the interventions derived from research. Yet there was no real agreement about what ‘long-term’ means precisely or, as a result, when epIA-AR should be carried out. Discussion also indicated that the time lapse between the completion of a research project and undertaking the epIA-AR is likely to influence reported impacts.

In relation to which, several participants emphasised the need to assess the impacts of agricultural research on both economic and non-economic dimensions. Some participants described initiatives they had been involved in and the approaches they had used to assess non-economic impacts. The strengths and weaknesses of using the sustainable livelihoods approach for epIA-AR (Adato and Meinzen-Dick, 2007), encompassing both economic and non-economic impacts within a single holistic framework, were described by several participants.

Three specific methodological issues were also debated in some detail during the conference. The first concerned the merits of social network analysis as a tool in epIA-AR. Participants were divided in their opinions about the value/practicality of this approach. A second issue was the validity of propensity score matching in dealing with potential ‘selection bias’ when farmers themselves decide whether to adopt a given innovation. A number of alternative quantitative methods were proposed as alternatives. Thirdly, there was a discussion about the appropriate criteria for selecting case studies for epIA-AR analysis. Finally, several participants in the conference expressed concern about lack of resources and capacities for epIA-AR and the perceived lack of importance attached to it by researchers and research institutions.

Data on Agricultural Research Activity and Expenditure

IMPRESA collated data on agricultural research activity and expenditure from 20 individual country reports, as a prerequisite for agricultural research impact evaluation. Official statistics on aggregate volumes of research expenditure, at both European and national levels, are only intermittently available and not fully compatible. IMPRESA assessed the available official data and supplemented this analysis with secondary data and interviews with key informants. This established the best possible estimate of recent levels and trends of expenditure on agricultural research. The expenditures the 20 European countries (19 EU Member States and Switzerland) analysed are thought to account for around 95% of European agricultural research expenditure.

Standard economic indicators for measuring expenditure on research and experimental development are set out in the OECD’s ‘Frascati Manual’ (OECD, 2002). This classifies gross expenditure on research and development (intramural expenditure) across four sets of actors, government, higher education, business and enterprises and private non-profit. There is a threefold classification of research activities: on the basis of fields of science, by economic activity, and by socio-economic objective. To arrive at consistent and comparable data, this information from all four sets of actors should be collected using the same classification framework. In practice, however, governmental, higher education, private non-profit data mostly fall under fields of science or socio-economic objectives while business enterprise data is mostly by economic activity. Data on governmental budgetary allocations are also available for socio-economic objectives, but these do not necessarily reflect actual expenditures.

Much effort goes into harmonising European statistics in order to ensure the quality and consistency of data. Most countries have a national statistical office or institute that collects and publishes data and forwards them to Eurostat. In some countries, ministries responsible for research also supply relevant statistics. Nevertheless, there are differences in the levels of detail about expenditure on agricultural research between countries. In some countries this data is fully available. In others there are significant gaps in the data. In the Netherlands, Sweden, Switzerland and the United Kingdom less than 25% of data about such expenditure is publicly available on Eurostat’s website. Classification according to socio-economic purpose is poorer, containing even fewer values than those collected by fields of science; five countries provide no data at all and, of 20 countries studied, 14 reported less than 50% of the possible total.

The main gaps in R&D expenditure data relate to businesses and enterprise. These data are reported separately by economic activity, and fall into two main classifications: agriculture, forestry and fishing, and the manufacture of food products and beverages. Less than half of this data is reported to Eurostat. Some countries do report business R&D expenditure but in formats that are not compatible with European reporting protocols. In addition our interviews raised concerns about the accuracy and breadth of the Eurostat’s private sector data.

Another way of approaching the issue of agricultural R&D expenditure is to examine the global budgetary allocations for government-financed research that has socio-economic objectives. Here we found almost complete data from 2007, but this data only relates to budgetary provisions, rather than actual expenditure, and the two can differ substantially.

The Eurostat database coverage of data on agricultural research expenditure is often inconsistent. Most commonly it is classified by field of science. As noted, there is more information available about budgetary allocations than actual expenditure, and the two do not always match. This situation should improve in the future as a result of a Commission Regulation of 2012 obliging member states to provide data, broken down by discipline, on higher education and government research expenditure, on at least a biennial basis. However, this Regulation has yet to work its way into the system and IMPRESA was unable to access such information.

Notwithstanding these caveats about incomplete and incompatible data, IMPRESA teams identified five countries that appear to be the main investors in agricultural R&D: Germany (22% of the estimated total for the 20 countries), Spain (17%), United Kingdom (12%), Italy and France (10% each). Together these five counties accounted for more than 70% of identified agricultural R&D expenditure between 2008 and 2013. During this period budgetary appropriations for agricultural R&D declined by 7% (in current prices) across the EU as a whole. These data should be treated with caution as several countries that reported an increase in budgetary allocations actually experienced an ex post decrease in spending.

Four countries with the largest public expenditures on agricultural R&D (an estimated 61% of the total expenditure in the EU28), provide consistent data on agricultural R&D expenditure, broken down into governmental, higher education and business enterprise (France is the exception in that it provides data in a format which is not comparable to that used in other countries). Between 2008 and 2012 total government spending on agricultural research in the four countries rose by €71 million (up by 4%), higher education spending fell by €117 million (down by 12%), and business enterprise expenditure rose by €12 million (up by 6%). In terms of total GERD undertaken by all sectors, overall research spending fell by €34 million (down by 1%), although taking inflation into account means that the real reduction is greater. In countries with lower levels of public sector expenditure, R&D expenditure by business and enterprises increased over this period: in France it rose from €103 to €157 million, and in the Netherlands from €64 to €184 million.

To supplement official data, IMPRESA also collected secondary data and expert views from the 20 review countries. Eight of the twenty countries reviewed are believed to have increased their agricultural research expenditure between 2008 and 2013, including Germany, where there were significant increases in research funding from both the higher education and the government sectors. Seven countries, including the United Kingdom and Spain, decreased their expenditure, the latter as a result of significant restructuring of the public agricultural science budget bought about by substantial cuts in overall public expenditure. Five countries, including France and Italy, maintained their expenditure at approximately the same level. It is expected that France will maintain its agricultural research expenditure at around the same level in the foreseeable future. In Italy, agricultural research is beginning to be affected by reductions in overall public expenditure, a reduction which may be partly offset by increased private sector investment in research into agriculture and agri-food aimed at developing a stronger bio-based industry.

In most countries the number of people employed in agricultural R&D is in decline, largely as a result of public sector budget cuts. This trend has even occurred in countries that have increased their expenditure on agricultural R&D. There has been a rise in the number of short-term contracts, and few informants anticipate that the numbers of permanently-employed public-sector agricultural researchers will return to former levels. This raises issues about the stability and critical mass of agricultural research expertise in Europe.

In the public sector, research funding is increasingly funded through competitive tendering and research institutes are receiving less core funding. At the same time the private sector is engaging more in collaborative research with public research organisations and is playing a stronger role in shaping research agendas. Multiannual research programmes are also of growing importance in shaping the research agenda, often focusing on environmental issues, such as climate change and energy efficiency. Key informants identify an increasing focus on promoting innovation and making technological transfer activities more robust and less spending on basic science. The EU’s FP7 and Horizon 2020 programmes are important influences on the agricultural research agenda.

The framework of agricultural science in Europe is in flux, partly as a consequence of public austerity policies that have been adopted since 2008. Public-private partnerships have stimulated some additional research activity, but this is probably not enough to compensate for the decline in public funding. A refocusing of objectives is apparent, with a shift from science-driven to innovation-driven research activities. While an economic case can be made for public research when market failures exist, the justification for subsidising near-market and commercial driven research is weaker. Enhanced farm productivity can improve supply chain efficiency and may result in more value being created, but it does not necessarily resolve issues about the distribution of that value across the chain, and who ultimately benefits from research subsidies.

This investigation provides a rich source of material about the current state of expenditure on agricultural research and innovation in Europe. However, it is weakened by incomplete and only partial details about the levels of expenditure and personnel and the capacities and strategies in both the public and private sectors. More comprehensive data would lay a better foundation for more clearly identifying the impact of agricultural R&D, whether or not it needs expanding, its orientation and mix of responsibilities between those engaged in agricultural research activities.

Four main conclusions can be drawn from this investigation:

− generally partial and incomplete data hinders the EU and governments of Member States from developing effective evidence-based policies;
− despite gaps in the data, it is clear that agricultural research expenditure in Europe has declined in both real and relative terms, so the need to have an accurate and comprehensive understanding of the state of agricultural R&D as a whole has become more urgent;
− integration of private and public funding offers some scope for more effectively targeted activities, which may partially compensate for the overall reduction in spending volume; and
− changing levels and mix of funding for agricultural research are affecting institutional structure and orientation, as public-funded are diversifying into commercially-funded work and as emphasis shifts from basic to applied and developmental work, research priorities are increasingly being shaped by multiannual strategies.

Following completion of IMPRESA country-level analyses of trends in agricultural research expenditures, and faced with the gaps in data availability, a workshop organised by FAO and IFPRI in Rome in April 2015 discussed and formulated recommendations for improving the future monitoring of agricultural research in Europe. Three feasible options were identified:

A. to improve the collection of R&D statistics (the most exhaustive approach)
B. to conduct an annual survey of public research organisations based on the Farm Accountancy Data Network (FADN) model (intermediate approach)
C. to produce an annual report by member states on research investments (short-term approach)

Following the workshop and subsequent consultation with decision-makers at EU and national levels, improved recommendations were finalised in November 2015.

Option A would consist of an update of the existing institutional framework for the production of research statistics in Europe, obliging all Member States to provide data on an annual basis, using the same categorisation for all sectors of performance (government, higher education, business enterprise or not-for profit) to ensure a common standard of comparison. This would require an update of the relevant Regulation by the European Commission and European Statistical System Committee and would apply to all organisations officially responsible for the collection of research statistics.

This option would have the advantages of developing a new statistical infrastructure, providing exhaustive data sets, and identifying alternative sources of data. It would enable objective analysis of the policy relevance of European agricultural research expenditures. However, its implementation would be expensive and would not assure data quality, as gaps based on national collection methodologies may still occur. Option B offers an opportunity to tackle this issue. Option A could be implemented through:

• development of a new methodology for an exhaustive, annual data collection exercise, prioritising mandatory data, guaranteeing their final quality and harmonising the classifications used, which could be based on the existing instructions and regulations; the joint OECD/EU manual for statistics on innovation (the Oslo Manual: OECD, 2006) should be revised within 2 years (with Eurostat taking the lead) to allow a tighter focus on innovation and impact assessment such that the revised methodology in the manual would provide a new definition of ‘innovation’
• introduction of a new classification category that includes bio-economy aspects and of a specific meta-classification of agricultural R&D (following the example of the International Energy Agency)
• data collection sponsored by leading stakeholder institutions
• pilot implementation of the above recommendations in volunteering countries

Option B would broadly involve emulating the FADN methodology by building a European network responsible for collecting data on research expenditure and activity from those who perform the research. FADN currently surveys the business operations of agricultural holdings in order to evaluate the impact of the Common Agricultural Policy. This network could be created by DG-Agriculture, and would involve the major public research organisations from the agricultural sector.

This option would have the advantage of establishing a harmonised cross-national methodology. Research establishments would be motivated to provide their data by the offer of providing feed-back from the survey results. This option could also be used to collect private-sector data, through observation of the increasingly common public-private collaborations. However, the human and financial resources required to implement Option B would not be much less than those required for Option A. In addition the potentially sensitive nature of the information collected could be a constraint on the comprehensive collection of, and access to, data. Option B could be implemented through:

• data collection by nominated representatives emulating the FADN model
• accurate definition of the main end-users of statistics and of their needs and building a methodology specifically designed to meet these needs
• data collection be carried at reasonable and regular intervals (say every 3 to 5 years)
• encouragement of the interest and involvement of researchers in the process by explaining the data needs during a workshop prior to data collection, and providing feedback on the results in a subsequent workshop
• support from public sector agricultural R&D institutes which could provide insights on private sector agricultural research activity from their engagement in public-private partnership activity

Option C would consist of an annual review of agricultural research expenditures at the Member State level. It would provide qualitative insights on research investments (such as performance and innovation adoption at farm level, the relevance of research topics and public-private partnerships). This review could be prepared by the Ministry of Agriculture or the Ministry of Research in each Member State, and delivered to the European Commission, or to the Joint Research Centre.

This option would have the advantage of being less demanding than the two previous options, while systematically and regularly providing important qualitative intelligence, essential for monitoring prioritisation. However, ministries asked for information might consider this as an audit of their own services, and this option would not, in itself, be sufficient to generate comprehensive data for rigorous monitoring purposes. Option C could be implemented through:

• annual reviews conducted at the Member-State level that would meet four criteria: representative sampling; clarity; standardisation; and cost efficiency
• provision of clear categorisation of disciplines that are located at research field boundaries (such as molecular biology)

This option would provide learning opportunities from cross-country comparisons, and could also be combined with Option A and/or Option B.

Choice between these options is highly dependent on whether one adopts a short or long-term view. On a short-term basis, Option C offers a relevant and systematic approach with a lighter implementation process than Options A or B. However, data collected through this approach would be mainly qualitative and might not be sufficient to allow a proper evaluation of the impact of agricultural R&D. It would need to be done in combination with one of the two other options, more likely option A, to provide a meaningful basis for any analysis. On a medium-term basis, Option B seems the most suitable strategy to collect comprehensive data related to the public sector. Research organisations are in a better position than ministries to provide detailed information on their investments as they have all this information available in their accounting databases. A combination with Option C might give a more complete picture. On a long-term basis, option A emerges as the most accurate and sustainable approach for the collection of relevant statistical data that can be compared over time in order to assess agricultural research investments. Quantitative data could be complemented with qualitative information gathered through Options B and/or C. To be made more complete, this option could also include the consideration of additional and alternative information, such as for example, administrative information.

Each of the options would involve conducting questionnaire-based surveys. To make these questionnaires more efficient than what is already in place, there is a real need to target end users’ needs; design a clear and appropriate methodology, including precise instructions; persuade data providers of the importance and relevance of collecting and analysing these data; make collection obligatory (which would require legal changes); provide feedback to the data providers when the study has been concluded; include the collection of qualitative data; and allow an adequate time frame for collecting and analysing data.

Based on potential combinations of the three options described above for collecting quantitative (Options A and B) and qualitative (Option C and also Option B) data, the new methodology should (i) enhance the transfer of knowledge and (ii) be adapted to the outcomes and impacts policymakers wish to achieve in the short, medium and long term. The new methodology would improve our knowledge about investments in agricultural R&D in the public sector and could later be extended to the private sector.

Case Study Investigations of Science-Based Agricultural Innovation

IMPRESA conducted a comparative analysis of six individual case studies of science-based agricultural innovation. The aim of this approach was to develop and test a methodological framework for assessing the impacts of agricultural research, analysing, in depth, innovations and research projects or programmes, to explore the complex processes that occur along related impact pathways. The case studies were selected on the basis of scale and agro-ecological and socio-economic diversity, and the theoretical framework adopted was based on an Impact Pathway Analysis (IPA). Conventionally, IPA has been used as an ex-ante approach, prior to implementing a research programme. It is a causal model that summarises the way in which the innovation pathway is intended to occur, from the implementation of research activities to the outputs, outcomes and impacts achieved. However, in the IMPRESA case studies, the goal was to carry out ex-post evaluations of the impacts and role of the research.

Case study investigation introduced a participatory element into IPA, to involve stakeholders more in the evaluation process and to increase the likelihood of them using the evaluation results for improved current and future implementation of research programmes. We employed the Participatory Impact Pathway Analysis (PIPA) (Douthwaite et al., 2007) at the workshops to enable reconstruction of the innovation pathway. This approach was complemented by some additional methods. The ex post impact approach used Outcome Harvesting, examined the role of actor networks using either Social Network Analysis or Stakeholder Mapping, and triangulated the information collected from different sources in the course of the evaluation process, using both Process Tracing and asking counterfactual questions in semi-structured interviews with actors. The impact pathway model used in IMPRESA for assessing the impacts and role of the research seeks to capture not only the inputs, outputs, outcomes and impacts of agricultural research but also the way in which they interact through feedback loops and interactions between different technical, commercial and institutional spheres.

Each individual case study was implemented using the same sequential approach. IMPRESA developed a case study manual for this purpose, which sets out the following seven steps:

1) Initial screening: this step outlined the case study narrative of the innovation and its adoption phases, and produced the initial map of the actor network. It provided a tentative list of outcomes and impacts, joined in a pathway. An exploratory plan was also designed to address the main impacts to be investigated and the related research questions.
2) Stakeholder pathway: this step mapped the impact pathway from the point of view of the stakeholders. Several methods for doing this exist, and two possible variants were adopted: the ‘world cafe’ and the ‘outcome harvesting’ approaches. The ‘world cafe’ approach consists of launching the subject as an open-ended question into a focus group of stakeholders and asking them their views about the innovation process. The second method is more structured and the discussion more directed. Participants were asked to (a) identify the outcomes; (b) how those outcomes were achieved, and; (c) draw an impact pathway that links the research activities, outputs, and outcomes. On completion of this procedure, stakeholders were shown the impact pathway drawn by the research team to compare and contrast with their own joint conception and explore the reasons for similarities and differences between the two.
3) Refinement of the pathway: this step added theoretical elements to the stakeholders’ pathway and to fill the ‘links record database’ (identification of the types of links and mechanisms). This also helps to refine the exploratory plan in terms of the impacts and research questions to be investigated.
4) Data collection: In this stage data was collected for the Social Network Analysis and the impact pathway approach, through in-depth interviews with actors and stakeholders.
5) Evaluation of the pathway: the process tracing method allowed researchers to investigate theoretical explanations that confirmed (or refuted) each link. This involved examining whether the conditions are necessary and/or sufficient and if any alternative explanations existed. The more convincing the alternative explanations were, the more the link was called into question.
6) Feedback round: stakeholders were offered another opportunity to provide feedback on the evaluation of the research and innovation complex in a final focus group.
7) Conclusion: final conclusions were drawn regarding impacts, provision of recommendations for public policies, and reflection on what was learned in applying the method.

While the original case study manual provided a menu of options for carrying out individual impact evaluations, experience in using it showed a need for more flexibility in order to cope with the very wide range of cases that were studied. After implementation and review, a revised version of this manual has been developed and is available for use in future evaluations.

A relatively small number of six case studies, in five different countries, was chosen in order to allow detailed and in-depth comparisons. They cover a wide range of agro-climatic, socio-economic conditions and different sectors. These include:

• Organic production in Camargue (FR)
• Integrated Pest Management in olive production in Canino (IT)
• On-farm biogas in Tuscany (IT)
• Dairy cow fertility index (UK)
• Optical crop sensor for precision farming (DE)
• Varroa control product in beekeeping (BG)

In these case studies, a number of key features can be identified. The research initiatives were mostly oriented towards improving the economic performance of farming, or towards solving environmental issues related to farming practices, or, in the case of the UK, ensuring the survival of the industry. The objectives outlined in the research proposals spoke in detail of expected outputs and outcomes, but there was little if any information on the expected impacts. Nevertheless, in most cases expected, or hoped for, impacts can be derived as plausible consequences of the objectives.

The six case studies were very diverse in terms of the activities, outputs, outcomes and implied impacts of agricultural scientific research. Nevertheless, all yielded evidence that their intended (if implicit) impacts were met, at least to some extent. In all cases there were beneficial impacts at both the farm and territorial level. In some instances a number of unexpected indirect impacts were discovered, many of which were either negative or ambiguous (the emergence of a black market for reselling the subsidized Varroa control product, and an intensification of dairy systems, are just two examples).

Most case studies contained some elements of scaling-up. Typically, the scaling-up was linked to capacity building, publicising the results of the research, setting up lobbying or marketing organizations, making changes to the regulatory framework or making the new product or technology easier to use.

In all the cases, the research took place in a changing institutional environment. Sometimes these changes preceded the research. Sometimes they occurred while the research was being carried out and at other times after the research had been concluded. These included changes in governance, market conditions, the legal framework and available financial support.

Other enabling and disabling factors were taken into consideration. These included human and social capital, the relations between actors, resource availability, economic prospects, institutional and policy frameworks and advisory services. A variety of social factors, linked to the capacities of key actors, were found to foster the innovation process. One of the most important of these was trust between actors which fostered networks and collaboration, as well as contributing to development of the skills of participants. Economic factors also often played a prominent role. In all the case studies, the role of public advisory services was either limited or non-existent.

An objective of the case studies was to ensure the participation of actors and other stakeholders who were involved in the original projects. The ex post nature of the case studies made this difficult in some instances, especially where the original research had taken place many years previously and key actors or stakeholders had retired or had moved to different positions. Facilitation skills and expertise in participatory methods played a key role in the workshops as there was a need to cope with the tendency of some stakeholders to dominate discussions and to manage conflicts between participants. Given the importance placed on capacity building as an intrinsic part of the innovation process, agricultural research should involve relevant social scientists and professional facilitators from the design stage of projects onwards to promote effective participation by stakeholders.

In conclusion, it is important to embed a culture of impact across the entire applied research process. Recommendations made relate to the entire research cycle: designing and planning research; the process of research itself; the analysis of the research project’s performance (which should inform subsequent projects and programmes); and the institutional context in which research takes place. Of these, the recommendations relating to the initial pre-research phase are of paramount importance. It is much more effective to involve, at the outset of a project, the key stakeholders other than scientists or policymakers. Trying to include other stakeholders and their perspectives at later stages runs the risk that the research will be poorly targeted or of limited relevance. Other important recommendations include mandatory interim review and effective impact monitoring. These can also play an important role in ensuring that stakeholder engagement is effective rather than tokenistic, and ensure increased research relevance and impact. From a research policy perspective, innovation theory suggests that, while impacts of research and cannot entirely be directed by intervention and regulations, these play a crucial role in creating an enabling (or disabling) environment for innovation, in contributing or hindering to capacity building and in influencing access to funding. The IMPRESA comparative case study analysis highlighted four key areas as being of particular significance: strengthened support for agricultural innovation; improved engagement with the private sector; better coordinated policies for research-based innovation and agricultural policies; and improved availability and access to research data.

The recommendations all point to the need to acknowledge the important role of ‘soft’ factors in helping research achieve more impact, and the importance of widening the role of all stakeholders in the innovation system. Such approaches do constitute a challenge to existing power relationships and the existing innate conservatism inherent within the dominant log-frame approach to evaluation. It also raises wider questions concerning the collaboration between the public and commercial interests and the equitable sharing of benefits between them.

Quantitative Assessment of Impacts of Agricultural Research

Quantitative analysis in IMPRESA used different approaches to assess impacts of agricultural research expenditure that differ from each other by the scale of analysis and the specific approach used, but were complemented by an analysis of the main cross-scale issues.

IMPRESA investigated the connection, at EU country level, between public research expenditure and agricultural productivity. It first reviewed available information concerning expenditure on agricultural R&D and agricultural productivity and implemented a graphical analysis of their trends. Subsequently, econometric models were used to quantify appropriate lags of impact effects of expenditure on productivity and derived a rate of return of research expenditure.

While output (Gross Agricultural Production/ha) has been stable over time, the levels of inputs per hectare have decreased. This trend is measured using Total Factor Productivity data produced by Fuglie (2012) from FAOSTAT data. Examining yearly trends, the sample shows a notable variability of relative annual changes across countries, from about 1% for the United Kingdom to 4% for Denmark. Indeed, a deeper exploration of the yearly evolution at country level reveals that most countries record a flat trend until 1990 (for Italy, Norway, Portugal and Spain 1987) and a steady (variable across countries) increase afterwards. Only the Netherlands and Sweden show constant positive tendencies along the entire considered period. It is plausible to attribute this increase in TFP primarily to research.

The results of the econometric models constructed are consistent with the hypothesis of a relevant contribution of research to productivity increase. The time lag of research effects on agricultural productivity is estimated to range between 9 and 18 years, depending on the model employed. The same considerations apply to the derived estimates of the payback to public research, which show a value of Marginal Internal Rate of Return (MIRR) between 7 and 15%.

Beyond an informative discussion about the magnitude and implications of the results, the main message that can be drawn from this analysis is that the impact of agricultural research on productivity in Europe is positive. The result can be considered as representative at European level, given the countries included in the analysis. This information was not available before, except in a limited previous study analysing the return of agricultural research on ten EU countries across the twenty-year period up to 1993 (Schimmelpfennig and Thirtle, 1999).

The approach has been limited by poor data availability, on both input (research expenditure) and output (production). The main limitations concern the length of the time series available and the level of standardisation (comparability over time and space) of expenditure data. In addition, the results, in particular the MIRRs, need to be interpreted as an average indication of the impacts of agricultural research, although results for individual countries or projects could have considerable variability.

Even accounting for this, results encourage continued exploration and effort to improve understanding of the research impacts on productivity both at country and aggregate levels, particularly taking into account the diversity characterizing the research policies of each EU country.

Complementing this analysis, and adding increasing complexity, an assessment of research expenditure impacts on multiple dimensions (economic, environmental and social) in (a selection of) European countries was performed. It adopted a multistep strategy: an initial identification of the priorities of European agricultural research; an estimation of returns to government research expenditure on Total Factor Productivity, comparing the traditional method with an adjustment to productivity calculation that accounts for environmental impact; and finally, proposal for an integrated qualitative-quantitative approach which characterises the causal path connecting public and private research expenditure with multiple impact dimensions.

European agricultural research topics show similar patterns among countries, with economic and health objectives prevailing over the others. These include a growing attention to environmental and social impacts and a gradual reduction in economic research. The shares of research investments strictly connected to competitiveness represent only a proportion of the total budget, which may lead to an underestimation of the elasticity of productivity with respect to expenditure. Even if restricted to a (relevant) part of the overall research investments, these results convey relevant and useful information, given that ex post impact evaluation should take account of objectives and targets of the investment.

The computation of an adjusted measure of productivity accounting for environmental externalities indicates greater responsiveness of environmental efficiency with respect to productivity, similar growth rates in European countries, and no relevant trade-off between productivity and environmental objectives. Results confirm the positive effects of expenditure on both traditional productivity and environmental adjusted productivity.

The qualitative conceptual pathway linking agricultural research expenditure to multiple sustainability dimensions was converted into a quantitative framework, and Structural Equation Modelling was applied to test the hypothesized causal structure. Results suggest a heterogeneous effect of government and business enterprise research expenditure through different pathways, with the latter mainly contributing to increasing and consolidating supply conditions, whereas the former appears mostly responsible both for improvement of competitiveness and for supporting better quality of life in rural areas (health, income etc.). Public research has a more complex pathway to impact than private, particularly as the former has a relevant role in education and knowledge transfer and a broader set of objectives compared to the latter. Transmission of effects to society is strongly dependent on the type of research activity and the institutional environment.

The multiple impacts of agricultural research derive from alternative transmission pathways of research-generated innovation. In this sense, the implementation of policies and regulations fostering diffusion of social and technological innovation should be a priority in the European agenda. IMPRESA results suggest a complex role for regulation, and the lack of a comprehensive policy milieu to support innovation and promote agricultural and food system transformation. Overall, this analysis implies a need for an innovation “policy mix”, where a coherent set of policy instruments is designed for long term objectives, taking into account interactions among them.

Investigation of the transmission of technical change adopted a structural decomposition analysis of international input-output tables drawn from the World Input-Output Database (Timmer et al., 2015). Taking an intermediate quantitative approach between scales, this examined the interaction between sectors of individual economies, and also transactions across national boundaries. Five European countries were studied (Germany, France, the United Kingdom, Italy and the Netherlands), along with the United States. These countries are the foremost international agricultural science performers. The focus was on the agricultural and food sectors of the economy. Technical change was decomposed, linked to environmental performance in each country (exploring the effects of technical change on major greenhouse gas emissions, energy and water use), and further analysed through trade linkages between the six individual countries.

In five of the six countries (all except the United Kingdom), real gross output in Agriculture, Hunting, Forestry and Fishing grew between 1995 and 2009; in France, Italy and the Netherlands, real gross output in Food, Beverages and Tobacco grew over the same period, whereas in Germany, the United Kingdom and the United States it contracted.

The first level decomposition showed that, for individual countries, technical change in Agriculture, Hunting, Forestry and Fishing was better than the average of all countries studied. France is an anomaly, in that apparently greater amounts of inputs were required in 2009 than 1995 to satisfy final demand in this combined sector. This puzzling result may be explained by a shift in the composition of the aggregated sector, requiring a different and more intense input utilisation structure. For Food, Beverages and Tobacco, technical change was also better than the average of all sectors; in this sector France, technical efficiency deteriorated, although not as much as in the economy as a whole.

The second level decomposition of technical change in Agriculture, Hunting, Forestry and Fishing showed that, in Germany, the United Kingdom, Italy and the Netherlands, the largest contributor to improvement was the Food, Beverages and Tobacco sector. The own sector contribution was comparatively modest. Apart from the services sector, other sectors of these economies contributed little to technical efficiency in this sector. The United States displayed a different pattern, with services making the largest contribution to improved technical change in the sector, the own sector contribution was significant, but Food, Beverages and Tobacco was a source of inefficiency. The same decomposition in Food, Beverages and Tobacco sector indicated that the Services Sectors played the most significant part in improved technical efficiency, with mostly larger within-sector contributions than from Agriculture, Hunting, Forestry and Fishing.

Generally, above average performance of the Agriculture, Hunting, Forestry and Fishing sector depended on improved efficiency of input production by the Food, Beverages and Tobacco sector, but also partly on services. The relationship was not reciprocal, as the above average performance of the Food, Beverages and Tobacco sector mostly came from its own improved technical efficiency and that of the services sectors.

Decomposition was also applied to overall emissions and natural resource use by industrial sectors. Changes in the totals over time can be divided into (i) changes in emissions per unit of output; (ii) changes resulting from the efficiency of input use; and (iii) changes linked to levels of final demand for the sector’s output. These have been analysed for carbon dioxide equivalent (CO2e) emissions, energy usage and water consumption.

Generally, reductions in CO2e emissions per unit of gross output in agriculture were broadly equivalent to averages for the economies as a whole, though the food sectors were not as good in this respect. However, reductions in emissions due to technical change were better than on average for both sectors.

There have also been overall declines in the use of energy per unit of gross output on average across the six countries, although not in the agriculture or food sectors. However, the energy use linked to technical change declined much more than the average, and more in agriculture than in the food sector.

Relatively few sectors dominate water use: agriculture, utilities, chemicals and food. Agriculture performed well in terms of reduced water use per unit of output, but not as well as in the chemicals sector in terms of technical change. The food sectors performed poorly in changed water use, with negligible effect of technical change on water inputs (mostly from the agriculture sector).

As well as analysing each country separately, further analysis of all six as a unitary system has been undertaken. The same decompositions were used, but exports and imports between the countries were not, as previously, treated as exogenous, but as intermediate inputs distinguished by national origin. This allowed the international transmission of technical change to be separately identified.

Unexpectedly, the effect of imported inputs on technical change in agriculture and food sectors was rather small. In two countries the impact on technical change was small and negative (the UK and US), in three its effect was small but positive (i.e. input-increasing, in Germany, France and the Netherlands) and in one (Italy) the effects were mixed.

For the agriculture and food sectors, some countries played a greater role in international transmission of technical change. Imported intermediate inputs from Germany had input-increasing effects in France and the Netherlands, and from the Netherlands in Germany. In the food sector, both the US and the UK benefited from beneficial imported technical change from all other countries, but this effect was very small. The Netherlands also imported beneficial technical change from Germany which offset a domestic input-increasing change but the magnitude was also small.

Second level decomposition explored the distribution of these modest scale first level effects between sectors in other countries. Broadly, the same sectors that were identified as important nationally were also important internationally; and although imported sources of technical change in the agriculture sectors came mainly from the food sectors, food sectors did not benefit as much from imported agricultural technical change.

Likewise, international environmental decompositions had only negligible effects on technical change, apart from in the Netherlands. The main reason appears to be the high degree of international integration of technical change that the Netherlands, has compared with the other five countries included in this study.

Quantitative analysis of private R&D and agri-food company level performance drew on a corporate data set of 307 companies from agriculture and food-processing industries from the EU, US, Canada and Japan for the period 1991–2009. The size of inefficiency and the determinants of inefficiency of given firms were analysed against the frontier production function, which defines the maximum output achievable. This was based on application of Data Envelopment Analysis (DEA) with two step bootstrapping which allowed correction of the bias in (in)efficiency and generate unbiased estimates for (in)efficiencies. Data show that EU firms tend to be little smaller in terms of revenue, sales and number of employees than their Northern American competitors. However, the latter had a same ratio of net income/revenue and R&D as those from USA or Canada, whereas in contrast Japanese firms appeared smaller and less profitable. However, they were more inclined to undertake corporate R&D but with a lower financial commitment on average.

When looking at the drivers included in the analysis, estimated results showed that private R&D had a positive effect on performance of firms by being associated with higher efficiency. At firm level the results also pointed to decreasing marginal returns to private R&D. Public R&D levels also seemed to have a marginal, although statistically significant, impact on performance. Some subsectors, such as the beverage industry, tended to perform better than others. The estimates also showed significant differences of R&D's impacts on efficiency across world regions and food sectors. From a policy perspective, the results of this study suggested that if the aim is to leverage European firms’ productivity and innovativeness (e.g. as set out as smart growth priorities within the Europe 2020 strategy) emphasis should go beyond the generally favoured high-tech sectors.

The use of DEA, which is a widely applied approach in the literature to estimate firm productivity, confirmed its documented advantages. A prominent result was the fact that it allowed the analysis to be performed without having to make assumptions about the form of the production technology or function. It is a non-parametric technique and the approach does not impose restrictions regarding the number of parameters required. Moreover, it is flexible as it is simple to deal with a whole range of inputs and outputs, with inputs and outputs which can have units. However, using DEA for this exercise also implied an analysis with observations treated as non-stochastic. Each analysis was also very sensitive to outliers, therefore requiring a preparation of the data with some implications for the structure of the sample. The estimation of the common production frontier implicitly assumed that all companies have access to the same technology and produce under virtually the same technological restrictions. Although some aspects such as country variations could be captured with dummies, it was necessary to reduce the sample to a sub-sample of rather homogeneous countries or companies so to expect unbiased empirical results.

In turn, the sophistication introduced by the bootstrapping procedure had a twofold effect. It both corrected the bias in (in)efficiency estimates from the DEA and also generated unbiased estimates for (in)efficiencies in the truncated regression. The procedure permitted bias-adjusted coefficient estimates and also calculation of proper confidence intervals to estimate statistical inference. However, bootstrapping tended to affect the structure of the data, potentially generating other forms of bias through its over-manipulation. A possible alternative would be to develop an instrumental variable to control for this. However this alternative was not seen as operational in this case.

The deviations or inefficiencies that the model was expected to capture were key to the analysis, as this is the indicator of performance causally linked to R&D investments. With the DEA and the estimation of the production frontier, all deviations from the frontier were attributed to the inefficiency term, while some of them could have been due to noise which is difficult to distinguish from the prime effect under scrutiny.

Data availability remained a main constraint preventing in depth and more nuanced analysis of the implications of R&D on the firm performance. Consistent reporting of R&D by firms in given countries is still elusive and prevents a reliable estimation of the size of inefficiencies. Even with available data, issue subsist as aggregation of R&D investments for a given firm not distinguish by sector or activity to which they were allocated.

Going into higher detail would entail higher heterogeneity (type of R&D, inputs-outputs) coupled with scarcer data. Moreover, increased specification of this sort makes the use quantitative approaches such as sector-wide DEA difficult, and thus tends to favour more focus on a narrow industrial definition, or case study approaches, which become more relevant.

The final method applied investigated the link between the introduction of an innovation (and its connection with research) and farm benefits in terms of profitability indicators, namely production, cost, value added and quality. Specifically, this investigated how knowledge about an innovation, and the scientific research and specific studies implemented to develop it, influenced the adoption decision and subsequent farm performance. An ad hoc questionnaire explored the adoption process of farmers and of analysing the hypothesized connection between prior knowledge of research and farm profitability. Relying on primary data collected in the Bologna province, an econometric analysis was conducted in order to assess in primis whether and how the prior awareness of available innovations affected adoption decisions, and then estimated the impacts of such decision on farm profitability. The first fieldwork survey was conducted from November 2015 to January 2016, and was complemented by further investigation in November and December 2016.

The results indicated that prior knowledge, although not a determinant for the adoption decisions, triggers significant improvements in profitability, in terms of quality of production. The research findings suggested that a timely diffusion of well-brokered research outcomes might provide competitive advantages to more aware and active farmers.

In more detail, results showed the importance of innovation for a large share of farms, considering a time frame of 20 years. Most frequent innovations were in the field of mechanical innovations and innovation aimed at water-energy saving. This is consistent with the fact that mechanisation is a widespread issue across sectors, while water-energy saving has prominence stemming from current trends related to climate change (potentially emphasised in the study area).

Multiple innovations were frequent among innovators. Classical factors, such as proxies related to size, were among the factors that explain better the adoption of innovations. Motivations for innovation adoption were largely related to the combination of cost reduction and production increase.

The process of innovation development and adoption followed two main pathways: of self-development by farmers, and development by (mostly) private companies. There was often background awareness of research, but this seemed only rarely to lead directly to technology development, and even less to adoption. This may have also been connected to the prevailing technologies which were considered relevant in the area, which in turn required important steps in terms of "engineerisation" of knowledge and fine tuning in local conditions (including machinery set up and feedback from users).

In either case, the mediation between research (or research performance) and farmers had an important industry component. Knowledge of the pathway from research to the farm seemed to be associated with better performance. However, a research connection did not affect all aspects of farm performance; rather, it was more directly connected to more specific points in the grid used in this study, particularly product quality.

This study suggested a need to explore the co-existence and interplay among different innovations and different innovation pathways further. Also, the interaction between awareness of the technology development path and actual technology performance at the farm level is an issue deserving further investigation.

Looking at all of these quantitative results in comparative cross-scale and process-related issues, several potential cross-scale/country connections have been detected, though a full assessment of their significance was beyond the scope of the project. The most relevant, although difficult to measure, are:

• public-private expenditure (trade-offs or complementarities, as for food firms, as well as timing);
• connected farm-food industry innovation patterns;
• spill-overs and decomposition of overall economic effects;
• farm-level corroboration of impact from research to performance.

The quantitative insights derived from the exercises also yield a number of interesting insights about the process leading to research impacts and how these processes are changing over time.

First, the range of objectives of research, both public and private, is expanding. This also implies that expected impacts are increasingly differentiated. Beyond the environmental focus of the last 20-30 years, this also involves different strategies connected to farm level management (e.g. yield vs. quality improvement, vs. resources conservation).

Second, intermediate steps between research and technology development are also more varied, considering the growing number of intermediate actors/actions, which makes more difficult to trace effects. Also, enhanced information and communication technologies are providing a number of new ways through which research findings are disseminated.

With evolving farming systems and more aware and connected farmers, the process between technology availability is also changing, with a greater role for better networked and more informed farmers in developing their innovation, and for industry in mediating knowledge increase with technology development.

An overwhelmingly important point is the timing of impacts. While the classical literature identifies lags of about 30 years from science to impact, micro and macro data in this project both suggest a time lag of between 10 and 20 years before the full impact of research is achieved.

In general, these quantitative evaluation exercises verify positive impacts of public research on productivity as well as of private research on performances of the food sector. Effects of research on productivity are mediated by the importance of innovation mechanisms for its transmission at farm level and are amplified by inter-sectoral spill-overs. Also the study shows the importance of multiple objectives for research, which also translates into multiple effects. The study has been able to qualify these effects, in particular through detection of impact variability and lags, decreasing marginal returns to private R&D, asymmetric transmission of technology change and heterogeneous impacts on different aspects of farm performance (e.g. cost vs. quality). These messages are largely consistent across scales.

The exercise highlights the strengths and the limitations of quantitative methods. Strengths are connected to the ability to numerical quantify the effects of research. Limitations are due to both limited data availability and difficulties in proving causality among the different variables considered.

Qualitative and quantitative approaches are largely complementary in building an overall picture of research impacts. The results lead to two distinct implications for policy. In terms of research policy, the main message is that research funding is generally justified; the levels of productivity detected, though favourable compared with returns from most sectors in the current economic conditions, are not high enough to suggest an “indiscriminate” push for more funding but rather suggest a careful understanding of priorities for investment in research in the sector and a cautious selection of mechanisms able to ensure focus on projects with highest net value added.

Improved evaluation instruments supported by regular studies are also required. To support this, more systematic data collection is needed especially in the direction of more analytical ability to connect funding with different research objectives and to better understand intermediate steps in the process from knowledge production to impacts (output, publications, etc.).

Institutional Support and Evidence-Based Policy Recommendations

IMPRESA-conducted research identified recommendations for both the conduct of research itself and for the enhancement of agricultural research policy at national and European levels. The primary recommendations were as follows:

Recommendations to Researchers:

Data and observation on the research process

• The scope of agricultural research is broadening and tackling social and environmental challenges rather than focusing on productivity. Classification systems must evolve to reflect these changes.
• Information on detailed dimensions of agricultural and food research could be gathered at low cost through a minimum set of recording standards adopted on a European basis.
• These could be aggregated into periodic reports at Member State level which provide quantitative and qualitative information on the scope and value of public agricultural research activity.

Developing a Culture of Impact

• A focus on motivation for impact could be achieved at three levels:
o planning for impact at the outset of research projects, using ex-ante impact pathway mapping;
o reviewing progress and impact pathways in the conduct of projects and programmes to taking new opportunities for impacts into account;
o keeping researchers in touch with networks of actors in the AKIS to provide feedback on the outcomes and impacts their work produces.
• While the impact pathway approach has strengths, due to its limitations (especially resource requirements) it should be used sparingly.
• More flexibility in competitive calls will allow responses to changing circumstances, especially if private sector involvement can bring new perspectives and changed market or policy conditions could improve potential impact.

Signposts for future research

• Mixed research methods hold most promise for tackling the complexity of knowledge and information systems.
• Investigation of impact must gain more attention than it currently receives, as public resources for agricultural science come under increasing budgetary pressure.

Recommendations for improve the quantity and quality of impacts from public support of scientific research on agriculture

Better targeting of research

• Improve research efficiency by encouraging a culture of impact among the scientific, extension and end-user communities through more participation in design, monitoring and evaluation.
• Enhance understanding of the productivity of research by undertaking 5-yearly reviews of its aggregate economic and environmental impacts and the average lengths of lags between expenditure and impact.

Enriched quality of agricultural research investment monitoring

• Enhance and extend frequency of data on agricultural R&D, with more and better information on private sector research, capture of bio-economy aspects and engagement with leader stakeholder institutions.
• Provide qualitative information on research investments to identify trends in topics and help to coordinate cross-country priorities and efforts.

Making use of improved understanding of impact

• Indicators are not suitable for evaluation because agricultural science is heterogeneous and its pathways to impact are complex. Also, ex-post Participatory Impact Assessment is expensive, time-consuming and, due to lags, comes too late to be of practical use. However:
• For Framework Projects at least (and recommended for Member State governments):
o Sample projects for investigation of impact pathways;
o To augment culture of impact, develop protocols for collection of data on impact pathways at proposal, review and evaluation.
• Create a code of practice for public private-partnership interaction in research and innovation, by identifying independent and professional innovation brokers.
• Develop datasets for proxies of intermediate outcomes and impacts.
• Make research data freely available and accessible.

Potential Impact:
General impact of the project

IMPRESA has produced new knowledge on the situation of agricultural research in the EU, with comprehensive illustration of research programmes and expenditure trends in 20 countries representing 95% of total effort from 2008 to 2013. This activity has led to widely disseminated recommendations among key actors in research policy, at regional, national and EU level that will enable programme owners to identify possible synergies between existing programmes and to avoid overlaps. Overall, the results contribute to an improvement of the effectiveness and the efficiency of future research programmes with the view to enhance the impact of public and private investments in agricultural research.
IMPRESA’s in-depth case studies and quantitative analyses have provided comprehensive insights into the mechanisms and processes whereby inputs and activities are translated into impacts.
In the case study work, its participatory nature has created networks across the impact pathway that can build on a new framework for the measurement of the impacts of research, as well as distinguishing case-specific issues which need to be taken into account in qualifying and controlling the use and adaptation of research results. This has improved understanding and appraisal of the short-term impacts of agricultural research.
New and improved application of econometric analyses to the connections between research expenditure and productivity, the environment, pollution and biodiversity, and complementary investigations of induced technical change in agri-food industries, at farm level, and transmission to the general economy have established a foundation for monitoring the longer-term impacts of SRA activities and enhanced evidence base for policy effectiveness.
Dissemination activities
IMPRESA activities have been undertaken in collaboration with a wide range of audiences. The IMPRESA Scientific Advisory Committee, which included 2 senior academics and 8 research professionals from international organisations and consultancies, provided extensive feedback on deliverables as well as participating in a number of IMPRESA-organised meetings.
Members of the project have attended periodic meetings to report on progress and results to the EU Standing Committee on Agricultural Research, and also to its Agricultural Knowledge and Innovation Strategic Working Group. Other activities involving wide audiences included:
• participants in an email conference on “Approaches and methodologies in ex post impact assessment of agricultural research: Experiences, lessons learned and perspectives”, which enabled the investigational approaches to be clarified, refined and extended in the first year of the project
• statisticians and research policymakers attended the international workshop in Rome on the topic of “Towards Better Monitoring of Investments in Agricultural Research in Europe”
• academics, policymakers and researchers who attended the final dissemination conference in Rome, which made presentations on the key insights emerging from the three main fields of project work: data and trends in European agricultural science; lessons of case studies of science-based innovation; and quantitative analysis of agricultural science impacts. A poster session provided more in-depth insights into the six case studies as well as quantitative impact analysis
• European Commission officials and other stakeholders who attended the policy workshop in Brussels on “Assessing the Socio-economic Impacts of Agricultural Research in Europe and Developing Countries”
The IMPRESA team involved 19 academic researchers and 16 professional researchers across 5 universities and 4 commercial and intergovernmental research organisations. In addition to the specific project-organised meetings identified above, scientific results from the project have been disseminated through reports, and shorter policy and research briefs, available from the project website. In addition, three peer-reviewed journal articles have either been published or accepted for publication, and IMPRESA partners contributed 15 papers and 1 keynote presentation to academic conferences, and also organised two symposia, one at the 29th International Conference of Agricultural Economists held in Milan in 2015 and the other at the 5th annual conference of the Associazione Italiana di Economia Agraria e Applicata. Overall several hundred researchers from across the world have attended conference presentations relating to IMPRESA research findings.
Future impact activities

IMPRESA partners continue to work to achieve impacts from the project beyond the grant agreement period. Our plan for increased impact includes continued engagement with the SCAR and its Strategic Working Groups, maintaining a world-wide web presence, and contributing to scientific meetings and literature.
• The coordinator of the completed project attends meetings of the Agricultural Knowledge and Innovation Strategic Working Group as an observer and will participate in future meetings relevant to its 4th Mandate point 6, to “analyze and discuss trends and evaluation systems and formulate indicators for interactive innovation in collaboration with OECD”;
• The IMPRESA website will be maintained, initially for two years as an active source of news and project information, and subsequently as a passive repository to provide access to archived project outputs; in particular, the IMPRESA PIPA Case Study manual, revised after piloting during the project, will continue to be freely available to interested researchers;
• 3 short films are available online, one in English (and subtitled) describing the overall project and case studies in Bulgaria, Germany and Italy; one in Bulgarian with English subtitles describing the Ecostop anti-varroa treatment; and one in German with English subtitles describing the Yara N-Sensor optical crop sensor for precision farming;
• Further conference presentations will be made in both academic and policy forums;
• More peer-reviewed articles will be published: a number of further articles have been submitted or are in preparation, including special sections for the Eurochoices journal jointly managed by the Agricultural Economics Society/European Association of Agricultural Economists and for Biobased and Applied Economics, the online journal of the Associazione Italiana di Economia Agraria e Applicata.

List of Websites:
All deliverables, and further information concerning the IMPRESA project, are available from the website at

Aberystwyth University (ABER), United Kingdom
Professor Peter Midmore (

Euroquality Sarl (EQY), France
Dr Cécile Fligny (

Food and Agriculture Organization of the United Nations (FAO), Italy
Dr Abdoulaye Saley Moussa (

Forschungsinstitut für Biologischenlandbau Stiftung (FiBL), Switzerland
Dr Matthias Stolze (

Institut fur Landliche Strukturforschung an der Goethe-Universitat (IFLS), Germany
Simone Sterly (

University of Bologna (UNIBO), Italy
Professor Davide Viaggi (

University of Pisa (UNIPI), Italy
Professor Fabio Bartolini (

Joint Research Centre of the European Commission (JRC), Belgium
Dr Sergio Gomez y Paloma (

Sofia University “St. Kliment Ohridski” (SU), Bulgaria
Dr Petya Slavova (