Final Report Summary - TRI-VALUE (Ex-post evaluation of Transport Research and Innovation in the FP7 ‘Cooperation’ Programme)
Executive Summary:
TRI-VALUE looks back into the work of FP7, Europe’s most important research funding instrument, on transport with the aim of per-forming an ex-post evaluation. The overall aims of TRI-VALUE are grouped into three pillars which are then linked to a set of five specific objectives:
• Assess pillar: (1) Measure concrete results of transport research funding in Europe, assessing positive and negative outcomes of research activities analyse implementation and management
• Evaluate pillar: (2) Evaluate quantitatively or qualitatively economic, social and environmental impacts of Transport Research and Innovation (TRI); (3) Measure the EU added value of transport research funding, notably assessing its contribution to foster excellent science, industrial leadership and to address the societal challenges; and (4) Measure the relation of the scientific and technical objectives in TRI programmes with the achievements in major EU policies, such as the Europe 2020 strategy, the Innovation Union Flagship Initiative and the ‘grand challenges’, and the White Paper on transport policy; seek to identify areas where further attention appears to be needed.
• Compare pillar: (5) Compare the performance of EU’s TRI with the approach in other major economies.
This information will be used to outline conclusions and recommendations for improving transport research and innovation.
The ex-post evaluation was evidence driven, i.e. a set of qualitative and quantitative information about the research and innovation work in FP7 was collected and critically analysed. This allowed the team to combine several perspectives on a single project, which improves the reliability of results and may correct eventual bias (e.g. by the coordinator who may be optimistic or by a reviewer with little contact with the project). The assessment was based on the application of the Research Impact Pathway (RIP), a methodology that helps to overcome some of the major limitations associated with these types of assessments, notably the fact sponsors of research want to assert impact soon after the research is undertaken whereas, in reality, many impacts will only develop gradually over (many) years.
The review performed in TRI-VALUE provides overwhelming evidence that research and innovation projects are producing a large quantity of relevant outputs. However, only a small share of projects delivered new services, new products, new norms and standards or patents, which may reflect the fact that TRI in FP7 is still focusing much more on basic research and early stage development rather than closer to market applications. It is still too early to say whether the project results will result in economic exploitation in many cases, however when looking at how the outputs are used and adopted by the relevant communities, results are less visible than the overall magnitude of the outputs production, with the research community emerging as the main user of the outputs produced. While most FP7 transport projects might have wider societal impacts in some way or another, in most cases those impacts will only arise in the long term and were not possible to quantify in TRI-VALUE. This a need to improve the measurement and quantification of project benefits and impacts and a new mechanism for following the impacts of projects after the official completion date (i.e. 5 years). In general, there is a positive evaluation of projects’ consideration of policy objectives in their work, in particular for GHG reduction and safety, and TRI-VALUE analysis has shown that transport research is responding to transport societal challenges, although in different ways across modes. There is still room for improvement regarding the links between transport research and innovation and the societal challenges; however, it is clear that there is a very good basis since there is already substantial relevant work taking place. TRI-VALUE also assessed the extent to which transport research projects are contributing to policy formulation. Results show that this is a major component of transport research and innovation.
TRI-VALUE evaluation concluded that the Transport Programme shows a large contribution to the achievement of the FP7 objectives and of the “Cooperation” specific program and reveals a strong European Added Value. FP7 Transport research also shows a good level of achievement of project objectives and expected impacts set out in the annual work programmes. Although multidisciplinarity was encouraged throughout the Transport Program, cross-cutting activities still lack an appropriate framework. Procedures for priority setting and Work Programme design are well established contributing to the realisation of ETP Strategic Agendas. Transport research had a substantial contribution to the formulation, implementation and assessment of Community policies and has provided initiatives to engage the public beyond the research community, in the debate on scientific issues and research results as well as in the field of scientific communication and education. This reflects the fact that transport research shows an adequate adoption of measures for dissemination, knowledge transfer and broader engagement. Regarding SME involvement, Transport research largely exceeds the threshold established for FP7. The collaborative research in transport allowed the creation of networks of excellence on transport research but, while the number of International cooperation actions supported has increased this cooperation is still limited. TRI-VALUE also concluded that transport research responds to transport societal challenges, although in different ways, and it produced positive impacts on innovative activity.
On the basis of the above conclusions, TRI-VALUE proposed the following recommendations:
• Improve coherence between research and transport policy objectives and establish guidelines/guidance on how to address them during the research and innovation cycle
• More emphasis on cross cutting issues
• Create favourable conditions for the uptake of transport research results
• Improve measurement of impacts and quantification of benefits
• Establish a Framework to assess the effectiveness of a Technological Innovation System
• Establishing consistent global strategies, addressing sustainability and competitiveness
• Focus international cooperation projects on network promotions
• Increase engagement of Civil Society organisations in research activities
Project Context and Objectives:
TRI-VALUE project performs an ex-post evaluation of transport research activities in Europe’s most important instrument for financing research, the Seventh Framework Programme of the European Community for research, technological development and demonstration activities (FP7). In addition to the ‘look back exercise’, TRI-VALUE ex-post assessment aims to learn from this experience and draw lessons for future European activities in this field. Accordingly, efforts are also placed at analysing how the experience of Transport Research and Innovation (TRI) in FP7 may be used to improve the next framework programme – the Horizon 2020.
Faced with an unprecedented economic crisis and increased global competition, the European Union (EU) has defined a strategy to support growth and job creation - Europe 2020 – aggregating seven flagship initiatives and in which boosting the knowledge economy and innovation is a key pillar. Accordingly, the ‘Innovation Union Flagship Initiative’, which aims to improve framework conditions and access to finance for research and innovation so as to ensure that innovative ideas can be turned into products and services that create growth and jobs, tops the list of such flagship initiatives. One of the key elements of the ‘Innovation Union Flagship Initiative’ was the setup of a new research and innovation programme for 2014-2020, ‘Horizon 2020’, which succeeds FP7. Horizon 2020 builds from the experience of previous programmes, thus reinforcing the importance of ex-post assessments such as the one performed in TRI-VALUE.
According to the Europe 2020 strategy the ‘Innovation Union Flagship Initiative’ will aim to re-focus R&D and innovation policy on the challenges facing our society, such as climate change, energy and resource efficiency, health and demographic change. This is particularly important for the transport sector, whose challenges summarized in the Transport White Paper touch upon all these areas. As stated in the White Paper, looking 40 years ahead, it is clear that transport cannot develop along the same path and to implement its new strategy innovation is essential. The six ‘Grand Challenges’ on which R&D will focus in the next programme can be summarized as:
- Global warming: In the current changing climate, how to better manage (prevention and recovery) the consequence of natural disasters such as flooding, fire forest, hurricane, dry area extension which tend to increase in numbers but also in intensity ?
- Tightening supplies of energy, water and food: In a constrained resource environment, how to improve the efficiency of consumption, the recycling rate while further reducing waste ?
- Ageing societies: As the life span of people increases, this raises numerous issues among which are economics, social inclusion, accessibility.
- Public health: How to provide medical care to everyone while minimising discrimination ?
- Pandemics: With the global and fast circulation of people and animals, disease can spread fast. How to improve the prevention and recovery in case of wide spread diseases since they potentially have a huge impact on the economy but also social stability.
- Security: How to improve the security of European citizens and their goods within but also outside Europe?
In summary, both European Transport policy and the Research and Innovation policy are facing transformational reforms. By looking back at the work on transport research and innovation in recent years TRI-VALUE will provide indications for shaping these reforms.
Horizon 2020 also introduces important changes to the research and innovation framework in Europe. Amongst others there is the commitment to focus resources on three priorities: Excellent Science, Industrial Leadership and Societal Challenges. Accordingly, TRI-VALUE will aim to help the European Commission in using the experience from FP7 to improve the way transport research is funded at the European level and in tailoring it to these new priorities.
The TRI-VALUE project objectives have as a starting point a project concept based on three pillars, schematically represented in Figure 1, and which will all contribute towards a main objective of providing recommendations for improving TRI framework in Europe.
The general objective of TRI-VALUE was further detailed, taking into account the project three pillars, for which specific objectives were identified.
Within the “Assess” pillar activities focus on the concrete measurement of the performance of TRI in the FP7 programme. The specific objective of this pillar is:
1. Measure concrete results of transport research funding in Europe, assessing positive and negative outcomes of research activities.
In the “Evaluate” pillar the data collected and the results of the assessment phase are further explored in order to look at actual impacts of TRI. It has the following specific objectives:
2. Evaluate quantitatively or qualitatively economic, social and environmental impacts of TRI;
3. Measure the EU added value of transport research funding, notably assessing its contribution to foster excellent science, industrial leadership and to address the societal challenges; and
4. Measure the relation of the scientific and technical objectives in TRI programmes to the achievements in major EU policies, such as the Europe 2020 strategy, the Innovation Union Flagship Initiative and the ‘grand challenges’, and the White Paper on transport policy; seek to identify areas where further attention appears to be needed.
The third pillar, “compare”, further extends this process of expanding the scope of the analysis, by focusing on the comparison of the performance in European TRI with the work developed in other economies. The specific objective of the “compare” pillar is the following:
5. Compare the performance of the EU’s TRI with the approach in other major economies.
In addition to these five specific objectives that may be directly related to the three pillars defined in the TRI-VALUE project approach there are two other specific objectives with a horizontal basis:
6. To identify a set of indicators to assess (1) the implementation and management, (2) the achievements and concrete results, (3) the economic, social and environmental impacts of transport research programmes and (4) the efficiency, effectiveness and relevance of funding; and
7. Provide targeted recommendations and conclusions for potential improvements in Europe’s TRI and in its practical implementation.
The ex-post evaluation performed in TRI-VALUE is evidence driven, i.e. a set of qualitative and quantitative information about the research and innovation work in FP7 were collected and critically analysed to attain the objectives stated above. The data and evidence base has been drawn from the data collection activities of the Project:
• Desk Reviews of the documentation for a sample of projects;
• Telephone interviews with a sample of Project Coordinators
• Online questionnaire responses from a sample of Project Coordinators;
• Face-to-face discussions with stakeholders at workshops and interviews.
The results collected through these various processes allow for several perspectives on a single project to be considered, from quantitative data collected from EC databases to the coordinators’ assessment or an independent evaluation by an independent project reviewer. This largely improves the reliability of results and may correct eventual bias (e.g. by the coordinator who may be optimistic or by a reviewer with little contact with the project).
Then a methodology to guide the assessment was used - The Research Impact Pathway (RIP) - first conceptualised for SITPRO (FP4 (Transport) evaluation) and further developed in the SITPROplus (for FP5-FP6 (Transport) evaluation). The thinking behind the RIP is that many of the impacts of a research programme have yet to materialise and that impacts will occur over a long time span, transcending the lifetime of individual research projects. The RIP provides a four-element sequence (production of outputs > dissemination > exploitation > impacts) for understanding how impacts can be expected to occur. Observations can then be made relative to this expectation for each individual project.
A key feature of the RIP is to record concrete evidence of progress along the Pathway and to avoid (sometimes wishful) speculation about what might happen. This may help to overcome some of the major limitations associated with the assessment, notably the fact sponsors of research want to assert impact soon after the research is undertaken whereas, in reality, many impacts will only develop gradually over (many) years. In this case this limitation was even more relevant, since although this exercise refers to an ex-post evaluation, from the 623 projects funded by the Transport program, only 336 projects (54% of the total) have been concluded at the start of the current evaluation. For the remaining projects, 110 (18%) were still ongoing and 177 (28% of the total projects) had their kick-off of activities after September 2012, date established as “cut-off” for early starters (i.e. projects with less than one year duration for which the assessment of outputs and results is hardly measurable).
Project Results:
TRI-VALUE project results were organised in two layers, the first corresponding to the production of programme level indicators, which focus on statistics addressing implementation of research activities and allow for an overview of progress in the development of transport research activities in FP7; and the second layer refers to the project level analysis, in which a detailed assessment of the results and impacts of projects is provided. Following the presentation of the main TRI-VALUE results for programme and project levels, the project conclusions and recommendations are described.
Programme Level Results
TRI-VALUE findings at the programme level, focus on the research implementation statistics: time to grant, budget breakdown, participation numbers and breakdown per type of participant, success rates and frequent collaborations. This data was exclusively obtained from the analysis of the EC database CORDA.
Time to grant
Time to grant, i.e. the time lag between proposal submission and start of the contract is a measure providing an assessment whether administrative simplification measures have been effective. A considerable reduction in the number of days to grant during the programme can be noticed: from the first to the last call in FP7, the time was reduced by 151 days (almost 3 months). To achieve this, the different simplification processes adopted, notably the electronic submission of proposals and negotiation files, were certainly large contributors.
An increase from 2007 to 2008 was noticed. This can be partly explained by the lack of a call in 2009, leading to some of the last calls in 2008 being delayed to start not until 2009.
Looking by sub-theme of the Transport program one can note that Surface Transport takes more time to grant than the other programs, in particular if compared with Aeronautics, where project size in terms of number of participants is in general comparable.
Figure 2 outlines the results regarding time to grant of FP7 transport research projects.
The FP7 transport budget
Total EC funding in transport calls over the FP7 period reached the 2.223 Mio €, of which:
• 52% Surface transport
• 45% Aviation (i.e. Aeronautics and Air Transport)
• 3% Cross cutting (horizontal activities)
About 75% of the EC funding in FP7 was allocated to four main objectives: Transport Greening, Efficiency of Transport System, Competitiveness and Safety & Security, present in the work program objectives. Modal shift and decongestion and urban mobility represented 7% and 8% respectively. Pioneering of transport took 5% of the budget while Policy Support took 6%.
This data supports the observation already made in the Intermediate Assessment that Policy Support and cross cutting issues were less covered in FP7, where there was a clear predominance of a modal basis.
Looking at the breakdown of FP7 transport average budget per project, it is possible to observe that budget for CPs (4.9 M€) is significantly higher than for CSA (1.1 M€), given the scope of activities under each of the funding schemes. Per sub-theme, it can be noted that budget per project in aviation (5.7 M€) is, on average, almost 1 million more expensive than the ones in SST (4.6 M€), for the same CPs.
In terms of budget breakdown per type of entity, more than 50% of the total EC funding (54%) is allocated to private sector participants (33% private companies and 21% SME). Research Organisations and Universities each get approximately 19% of the EC funding. Public Body participants take 5% of the budget.
Participations in FP7
During FP7, 2 679 proposals have been submitted corresponding to 29 584 applicants. Of those proposals, 623 were funded as projects, involving in total 8 302 participants. Table 1 presents the detailed breakdown of the number of proposals and projects per type of project and sub-theme.
Collaborative projects represent the majority of FP7 transport projects. 73% of the projects funded are CP and only 27% are Coordination and Support Actions. Since the FP7 programme was particularly targeted to address scientific and technological research, development and demonstration, this result is not unexpected. Cross cutting issues and horizontal activities, where the large majority of the CSA fall, represent a minor share compared with AAT and SST.
Figure 3 depicts the average number of participants per project and the type of participants, allowing for a more detailed analysis of the participations in transport FP7 activities.
It can be observed that, as might be expected, faced with wider scopes and budgets, the average number of participants is higher in CP than in CSA for both Aviation and Surface Transport. In the cross-cutting projects the average number of participants is almost equivalent between the two types of funding. For the type of participants it is particularly interesting to note the 24.5% share of participation by SMEs in FP7 transport projects, compared with a target of 15%.
Success Rates in Transport research
The success rate in the Transport FP7 programme is overall around 23%, reaching 30% for CSA but only 21.5% for CP projects. In all the sub-themes the success rates for CSA are higher than for CP; only for TPT (cross cutting activities) does the success rate presents a minor difference between the different funding schemes.
Regarding success rates per type of participants, the highest can be observed for Private Companies (34%), followed by Public bodies with 31%. Research Centres and Other show success rates around 30%. The average success rates for SMEs are 25% while for Universities it is 24% and 16.6% for Research Organisations. Success is higher in CSA than in CP, however it is noted that Private Companies have a high success rate (32.9%) in CP, compared to the other entities.
The same analysis performed only for the coordinating entities shows again that Private Companies are more successful in their proposal submission as coordinators (in average 31%). The success rate for Research Organisations as coordinators is nearly 25%, and is around 26% for public bodies.
Most frequent collaborations (Research Networks)
Figure 4 highlight the most frequent collaborations between Member States in the funded projects (ignoring collaborations within the same country). Overall, the strongest relations can be found among France-Germany-Italy-UK-Spain as the main players with the strongest pair being France-Germany.
Looking at each of the subprogrammes, i.e. aviation, surface transport and cross cutting, one can see some interesting differences. First, on aviation one can note a great project concentration in Germany and France, which happen to be the two most important Member States for EADS/Airbus. As for cross-cutting issues and SST Italy also emerges as a key player, sidelining with Germany and France.
Program Assessment by Project Officers (based on RESPIR)
TRI-VALUE also considered the assessment of respective Project Officers. Results highlight an extremely positive evaluation of the final results in the Transport projects in FP7, with just 2 in 227 cases, failing to achieve objectives and /or partly achieving objectives obliging to corrective actions.
54% of the 227 completed (and processed) projects fully achieved its objectives and 45% has achieved most of its objectives. Although 98% of the concluded projects available from RESPIR (227) fully accomplished and/or achieved most of its objectives, the assessment of outputs doesn’t show such a positive picture. Indeed, just about one fifth of the projects were identified by the respective PO as a “success case story” and producing an impact on EU policies. About 14% of the 227 projects are marked with “Good innovation potential” but only less than 8% (17 in 227 projects) were highlighted as with Substantial R&D breakthrough character.
Project level results
The TRI-VALUE results at project level essentially derive from the analysis of the production and use of project outputs and the attempt to link these to the steps of the Research Impact Pathway, which will allow conclusions on the impacts and added value of research activities. Moreover, the assessment of results took into account different yet complementary views, as an important component of the analysis was to evaluate whether there are significant differences amongst the transport modes, the project orientation, the budget size and starting date of the projects that would contribute to the results achieved. This is important in the sense that TRI-VALUE does aim to go beyond looking backwards to provide recommendations for future improvement of transport research in Europe.
This analysis revealed that orientation (i.e. whether it was a technology focused or a policy focused project) and mode are the variables that most affect the production of different outputs. Budget impacts particularly on the number of prototypes and pilot scale projects generated while starting date is a variable impacting mostly the number of publications.
Moreover, it is worth underlining that results obtained by TRI-VALUE were based mostly on the results of the online questionnaire to coordinators, referring whenever they differ significantly from those obtained from desk reviews or from Project Officers (i.e. RESPIR). The online questionnaire comprises the answers from 245 projects, representing approximately 41% of the total FP7 transport projects, giving a good level of representativeness for all the transport modes. Regardless of this high response rate, it is also worth noting that within TRI-VALUE multiple sources of evidence are used, allowing for a constant verification and assessment of data consistency.
The main results from TRI-VALUE’s ex-post evaluation are presented in six sections, with the first focusing on the production of outputs by research and innovation projects, the second analysing the dissemination of those outputs and the third looking at the exploitation of those outputs. The following three sections address the development of skills and networks, the contribution to policies and the EU added value of transport research and innovation. Finally there is a section that looks into the achievement of the programme’s objectives.
The Production of Outputs
The first component of the ex-post evaluation of TRI in FP7 looks at the concrete outputs generated by research projects as a whole. Figure 5 represents the share of projects generating a pre-defined set of concrete outputs, according to the online questionnaire.
As can be seen, a very high share of projects (>60%) delivered peer review publications – underlining the knowledge creation pillar of TRI – and of testing activities (validations and verifications) – which may be linked with activities related to development of new products or services. Development of software, tools, models and applications (not marketed) is also very common (i.e. occurs in more than half of the projects). However, only a small proportion of projects (<20%) delivered new services, new products, new norms and standards or patents, which may reflect that fact that TRI in FP7 focused much more on basic research and early stage development rather than closer to market applications.
Importantly, the type of output generated highlights significant differences, as expected, when comparing policy and technology oriented projects and the different modes. The outputs in terms of patents, prototypes, testing activities, models and tools, new technologies is considerably higher in technology oriented projects, as might expected; however, projects technology oriented also produced significantly more peer reviewed publications and PhD theses. Looking per mode it is noticeable that those producing more scientific publications in policy oriented projects belong to the aviation and water sectors, while in technology oriented projects it is rail and intermodal that do, with urban being the mode that maintains a high level of scientific publications both in policy and technological projects.
On the other hand, the number of policy oriented projects reporting the production of Assessments (i.e. impact assessments, consultancy studies, etc.), Handbooks & Guidelines and Policy Recommendations is greater than for technological ones. Per mode, it is interesting that production of assessments is particularly noticeable in water projects, while policy recommendations arose mainly from intermodal and road projects. However, it is in urban projects that this type of output is mostly frequently produced.
An interesting aspect is the fact, that while in general technological oriented projects were more market oriented, the proportion of projects in policy oriented projects reporting the production of new products (i.e. new materials, Physical Parts, Machinery, Facility Infrastructure) and Services (including ICT and non-ICT services) that have been introduced to the market is higher than the technology oriented ones, this being observed for both urban and rail projects.
When comparing for the same type of outputs, those that have produced more than 4 outputs (compared with those that have produced at least 1 output) consistently it is observed that, independently of mode, technological oriented projects (and typically those with longer duration and higher budgets) show the highest results.
Similarly important to the production of outputs is their use and adoption by the relevant communities. Respondents to the survey were asked to state which of the users they had named as important or involved in their research had actually made use of the research results, for instance referring to them in policy documents or by implementing them. Globally for the FP7 projects surveyed, the research community is the user that makes most use of the outputs produced (for about 40% of the projects) and implemented results for about 30%. For the other users this figure is about 20% of projects for reference and use in policies, a proportion that decreases when referring to the implementation of results.
However, as with the outputs produced, its actual use by the different users varies significantly by mode and project orientation. Coordinators were requested to refer to the uses of which they were aware; the main message is that for the majority of projects, coordinators are not totally aware of project results being used, particularly after the end of the project.
The use by EU institutions of policy projects is mostly noticed for rail and water projects, namely its use in policy papers and implementation (i.e. via a Directive or Regulation). For technology projects, the use of results is noticed mostly as “reference in documents”, namely for rail and urban projects.
Local and National Authorities use in particular the results from policy projects, this being visible most in urban projects, at least where coordinators are aware. This can be explained in part by the presence of more local authorities taking a role as participants in projects (i.e. CIVITAS). The use in policies and strategies is also noticeable in air projects, road and water. Coordinators for rail policy projects tend to be unaware of use of their results by local and national authorities, although they referred to their use in some technology projects.
As could be expected, SME’s made more use of technology oriented project results rather than policy, although the case of urban policy projects reveals a degree of use significantly above the other modes. This could be partly explained by the presence of several SMEs acting as consultants at local /urban level using the results of research projects frequently in other studies and projects. In technology oriented projects, implementation of research outputs by SMEs is particularly noticeable for the rail mode, followed by water and road, being slightly lower in air projects, where despite the presence of several SMEs as providers, the large companies still have a stronger role.
Regarding the use of results by large companies there is a trend towards higher use of technology oriented projects, perhaps with the exception of rail, where policy projects are more often used, referred to and implemented.
As noted above the research community is the main user of TRI project outputs. Here the difference between use of policy and technology project outputs is hardly noticeable, underlining the cross-cutting work of the research community.
Dissemination of project outputs
Dissemination is critical to increase the impact of project outputs. There are various ways in which project results can be disseminated: through publications, presentations at workshops or conferences, newsletters, website announcements or project reports. It is impossible to determine the best dissemination mechanism, as it depends on the “message” and the “audience”, accordingly, the TRI-VALUE online questionnaire addressed this issue by requesting respondents to rate the relevance and actual use of different dissemination means for the impact of their projects.
Coordinators were asked to classify the relevance of each dissemination mechanism used in a scale from 1 to 5, where 1 means “very low relevance” and 5 “very high relevance”. The results for the dissemination means classified as very relevant /very high relevance (4 and 5) are presented in Figure 6 according to their orientation and mode.
Globally, the “standard” instruments, i.e. most commonly used in transport research projects, such as websites and events, either other events or events organised by the projects, remain as the ones most reported by a high number of projects. Promotional material and newsletters are very relevant in policy projects and scientific publications for technological projects. Presence in social media is reported in nearly 20% of the projects, independent of their orientation, so it can be identified as a growing trend in transport research dissemination.
Results per mode reinforce the importance of website and events, especially in rail, urban and water policy projects and urban technology projects. Scientific publications are also identified as a very relevant dissemination mechanism for technology projects, especially at the urban level. It is also worth noting that newsletters, which used to be seen as the standard mechanism for TRI European projects are now ranked at values close to those of the brand new “presence in social media”, especially in technology projects. Another interesting element to notice is the pattern that air and road transports rely relatively less on Press Releases for dissemination, while urban and water seem to be making good exploitation of this mechanism both in Policy and Technology projects.
Based on the overall results one can conclude that dissemination is recognized as an important feature for European transport research projects, although different projects tend to use different dissemination mechanisms.
An important aspect to reflect upon is on whether the dissemination strategies put in place by transport research projects are succeeding in engaging the broadest possible public beyond the research community in the debate on scientific issues and research results. By looking at the results on questions on the use of project outputs by the different communities we can conclude that, since the research community is identified as the main user of transport research results and many project coordinators are unable to identify use being made of their outputs outside the research community, there is a lack of motivation for coordinators either to disseminate beyond the scientific community or to identify the use of their research results.
Still, several initiatives in the field of scientific communication and education, including the involvement, where appropriate, of civil society organisations or networks of such organisations, were launched over the program. Efforts have also been made towards targeting the future generation of researchers and professionals in the transportation area. Additionally activities for fostering coordination of research efforts (i.e. CAPPADOCIA, CORE-JETFUEL, REACT); capitalisation of results (i.e. SOLUTIONS, CAPITAL, TIDE), knowledge transfer (i.e. TURBLOG-WW, SEAHORSE, CO3) have been launched in practically all the transport sub-themes. Furthermore, specific actions to setup (i.e. ETNA) and give continuity (i.e. ETNA PLUS) to a network of National Contact Points to support and foster the active participation of stakeholders in European research calls and projects were promoted with large success in reaching a wide universe of stakeholders and in supporting activities.
Exploitation of project outputs
Dissemination and exploitation frequently come together in projects, notwithstanding their different objectives. Exploitation of outputs is market oriented, i.e. it looks to exploitation as a means to progress from the research results towards its deployment and uptake. Previous projects dedicated substantial attention to this issue and on-going projects are focusing on developing support to different consortia exactly on the exploitation aspects.
Exploitation of project outputs was assessed through a combined analysis of the online questionnaire and the interviews with coordinators. It is worth noticing that responses were generally slightly more optimistic in the answers received from the former, than from the latter.
Coordinators of surveyed projects were requested to indicate whether in the context of the project or after its conclusion, projects had developed i) a business plan; ii) direct contacts with stakeholders (i.e. users) or potential clients; iii) established agreements with private investors, and if there was any technology transfer associated with their projects. Figure 7 presents the results obtained from the online questionnaire.
Approximately half of project coordinators surveyed have engaged with developing a business plan and analysed markets. Nearly 90% of them are in direct contact with stakeholders and three quarters made contact with potential clients during the project period. Some of them (~11%) have even contacted with private investors such as venture capitalists. About 4% have succeeded in signing up agreements with private investors. About 40% of the respondents undertook technology transfer activities during the project period, increasing to almost 50% after the project conclusion.
Surprisingly, the views about economic exploitation of project results are more optimistic "during the project" than "after the project". Actually, within a conventional project cycle perspective, exploitation of results should be more interesting and promising once the project has concluded. It could be considered here whether these answers reflect some lack of commitment and interest about exploitation of project results by some coordinators, once the project is completed; this would indicate a significant barrier to implementation and a crucial failure within the innovation process.
Direct contact with stakeholders is highlighted as the more frequent action and result of a project from an exploitation perspective (93% of answers in the questionnaires and 87% in the interviews). Regrettably, the number of positive answers decreases significantly (to 76% and 59% respectively) once the project is completed.
Contact with potential clients is also pointed out by many coordinators as one of the exploitation-related activities undertaken. It is worth noticing that there are only small differences between the percentage of positive answers referring to the activity during the project and after the project. In the on-line questionnaire, these percentages are, respectively, 75% and 68%, and during the interviews, 62% and 40%.
The rest of the activities suggested by the questionnaire and the interview received much lower positive answers. The development of a business plan and/or market analysis nonetheless received a significant number of answers in the questionnaire (54% during the project and 34% after the project), but much lower ones in the interviews (31% and 12%). Technology transfers are mentioned by 30% (during the project) and 19% (after the project) of questionnaire responders, and by slightly lower percentages in the interviews (25% and 8%, respectively).
Contacts with private investors (VC, Business Angels...) and agreements with private investors, both during and after the project, are pointed out only by a small percentage of the responders both, in the on-line questionnaire and during the interviews.
Finally, it is worth noticing that no concrete figures were provided on the estimated market value of projects’ results. Actually, only two coordinators provided some detail (€300,000 in the case of one small CSA focusing on educational issues and an "extreme large value" (with no further detail) in the case of one rail project. These results provide some evidence about the need to dedicate more attention to the exploitation potential of project results, and the contact with key stakeholders for facilitating this.
The policy orientation of projects looks not to be exerting a major influence on the exploitation of results with the exception of contacts and agreements with private investors, namely in water policy projects. In technology oriented projects contacts with private investors are rarely indicated by coordinators in the online questionnaire. This could result from the presence of industry as part of the consortia.
The development of skills and establishment of networks
Projects often contribute to the development of effective cooperation and networks. The access to complementary expertise and “network” effect from FP7 projects is one of the most intangible assets generated by transport research. While the benefits of such assets are hard to quantify, having several informal or (formal) networks with members from different communities (research, industry, consultancies, NGOs) and countries is valuable. The involvement of junior researchers in projects is also frequently reported as an important contribution of research projects towards academic excellence.
In TRI-VALUE, coordinators were asked to indicate whether projects:
• projects led to the creation of a formal network under the scope of the project
• there was cooperation with other partners of the consortium after the project completion
• there was cooperation with other external organisations as result of the project´s activities
More than a third of projects have been successful in creating a formal network within the scope of the project, and an overwhelming majority of them (>86%) continued to cooperate even after completing the project for which they originally formed a consortium. Nearly 75% of the respondents also developed contacts with external organisations as a result of the project.
The analysis performed in TRI-VALUE showed that cooperation after project conclusion is mode independent and a good example of the networking effect of the collaborative research in the EU FP’s. In more than 80% of projects (exceptions are water policy and road technology projects, but still outline a high share of responses) follow-up collaboration occurred. As a result, collaborative follow-up with members of the consortium or with external organisations is also not really influenced by mode or project orientation.
However, transport mode and policy orientation seem to exert some influence in relation to establishment of formal networks, compared with technological oriented projects. While there are several examples of networks established from FP7 projects, this seems to be a particularly relevant result in water policy projects, with nearly 75% of projects in that category referring to this type of networking. In the opposite direction, for air projects, formal networks look to be a relevant result in technological projects, but none of the air policy project coordinators reported such a type of collaborative effect. For remaining modes, no significant differences relative to their orientation were noticed.
Contribution to Policies and Environmental, Social and Economic Impacts
The potential contribution to policies emerging from FP7 transport research activities was addressed by TRI-VALUE in the online questionnaire the interview and the desk review. In the online questionnaire, coordinators were asked whether their project contributed, directly or indirectly, to transport policy objectives. The comparison between the information provided by coordinators is generally coherent with the evaluation from TRI-VALUE desk reviews. Results are outlined in the Figure 8.
In general there is a quite positive evaluation among the objectives identified, in particular for GHG reduction and safety, followed by pollutants reduction and energy efficiency. At the other extreme, there was little attention to recycling measures.
On the energy side there appears to be much more focus on energy efficiency than on finding new fuels or energy sources, which may reflect the fact that so far regulations have focused on targeting directly emissions, which are closely associated with energy efficiency
Across all modes there is a large share of projects contributing directly or indirectly to those objectives.
It is clear that the modes with a higher contribution are Road and Urban, which may also reflect the high share of road-based mobility within transport CO2 emissions and/or the high regulatory pressure that was observed over the last few years.
At the other extreme, we have rail, perhaps because rail transport has a marginal contribution to CO2 emissions and is already among the most efficient modes of transport.
Policy orientation of projects exerts some influence in terms of a more direct or indirect contribution to the objectives, with technology oriented projects showing a more direct contribution, namely for emissions, noise and safety.
From the coordinators’ perspective, urban projects emerge as the ones with a direct contribution to the reduction of pollutant emissions, this being consistent for both policy and technology oriented projects. This fact can’t be disassociated from the fact that urban policy projects are aligned with the clean urban package objectives and EU initiatives such as the Covenant of Mayors. Technologically oriented projects on average have a more direct contribution to the objective than policy ones, as might be expected.
In general, projects show mainly an indirect contribution to reducing congestion, both for technological and policy oriented projects. The exception is urban policy projects with nearly 60% of the answers showing a direct contribution to this objective.
Security looks to be an objective not so strongly considered in transport projects or where projects’ contributions (either direct or indirect) are less evident; however the presence of a dedicated program on “Security” might partly explain this.
In what refers to Environmental, Social and Economic (ESE) Impacts, it is worth starting by underlining that while most FP7 transport projects might have wider societal impacts in some way or another, in the overwhelming number of cases those impacts will only arise in the long term. Accordingly, the results from this ex-post assessment shall be read with the limitations associated with the timing of the analysis in mind.
Indeed, topic descriptions in the FP7 working programmes routinely made reference to ESE objectives, and most of the FP transport projects may claim to have wider societal impacts in one way or another; however, in an overwhelming number of cases those impacts will only materialise in the long term. There is an inherent difficulty for coordinators (and also for evaluators) to substantiate how one particular research project can result in concrete ESE impacts.
In fact, it could be claimed that these impacts are primarily considered in the stages prior to actual research: firstly, by EC officials during the preparation of the topics in the working programme and secondly, by researchers during the preparation of their proposals. At both stages, there is some elaboration on how the research activity can contribute to policy objectives and produce particular ESE impacts.
However, no evidence was found about the active monitoring of these impacts during the deployment of research activities or at the project completion stage. This may explain the general lack of evidence found during desk reviews and interviews by evaluators regarding concrete ESE impacts from projects.
Macroeconomic impacts include effects of transport projects on macroeconomic variables (i.e. employment, consumer price, regional effects, international trade, and public authority budget). Therefore the indicators used for tracing these impacts were:
• Job creation after project development
• Consumer prices
• Differentiated impact on specific regions
• International trade
• Impact on public authorities’ budget
• Overall macroeconomic impact
Most assessment of macroeconomic impacts was based on the desk reviews, with the exception of job creation, which was assessed in three different ways:
• Through the on-line questionnaire completed by the coordinators (using a qualitative scale: no, yes- indirectly, yes-directly).
• During the telephone interview with project coordinators (using a quantitative 0-10 scale).
• During the desk review (using the same quantitative 0-10 scale).
Around 30% of projects contributed to job creation. The comparison of three ways to detect the job creation contributions shows a certain consistency. Nevertheless, few coordinators are aware that their projects create new jobs or improve the quality of jobs. This was an issue generally present in the approach of the European Technological Platforms with some differences among transport modes: ERTRAC and Waterborne platforms highlighted the problem of job quality in their sectors and they are looking for improving the conditions of their jobs and their attractiveness. Air emerges as the transport sector where more progress has been made in terms of improving job quality (e.g. the HUMAN project which analysed the design of aircraft cockpits).
In general, less than 20% of transport projects contribute to stabilize consumer prices and 16% seems to contribute to balance of differentiated regions. 30% of transport projects had impacts on international trade and 21% contribute to lower public authorities’ expenditures.
Impacts on key macroeconomic priorities at EU level (i.e. job creation, stabilizing consumer prices, stabilising public authorities’ budget and general macroeconomic growth) seem to be limited or, at least, not easily detectable in the transport projects.
Between 20% and 30% of transport projects were said to contribute to key macroeconomic priorities. However, coordinators think their projects have major impacts at macroeconomic general level.
Other economic-related priorities are addressed by a higher percentage of projects: 75% of relevant transport projects increase EU competitiveness and more than 50% boost the EU economy. According to the desk review, only 16% of transport research projects would contribute to improve the quantity and quality of jobs, in spite of the general contribution to "job creation" at the macro level claimed by 30% of projects.
In terms of environmental impact, the picture framed by TRI-VALUE analysis shows a modest direct contribution, although figures increase significantly when indirect contributions to environmental improvement are included. For example, over 80% of projects claimed to contribute directly or indirectly to reducing greenhouse gas emissions, 75% have contributed to improving the local environment and almost 50% to the development and use of sustainable/renewable sources of energy. To strengthen the environmental profile, better identification of the connecting paths between transport research and the environment, and more directly addressing environmental dimensions during the definition and selection of research topics could be helpful.
Direct social impacts associated with the FP7 transport projects were rather limited, outside the urban modes. Linking this to consideration of key social priorities at the EU level, few transport projects appear to address basic social services, poverty and social exclusion, although causal links are already known. Ageing is another domain European Transport Research could pay some additional attention to, as is also the case for security issues relating to user perceptions and attitudes; both are expected to further gain relevance in an ageing, multicultural society with a pervasive risk of segregation.
There appears to be a need to better identify the connecting paths between transport research and the social and environmental impacts, and to address these dimensions during the definition and selection of research topics. The implementation of a permanent tool for projects follow up (i.e. ”TRI research watch”) could support the ex-post evaluation.
However, the ESE evaluation undertaken by TRIVALUE suggests that the much needed socio-environmental transition mentioned in EU policy papers and ETPs' Strategic Research Agendas is still to be achieved.
An additional analysis would be needed in the future in order to gain deeper understanding of the barriers and possible pathways for increasing the short and medium-term ESE impacts of research. We suggest adopting in the future a more structured, interdisciplinary approach for the definition and assessment of transport research priorities and working programmes, including the various levels involved in the innovation cycle (niche-innovation, the prevailing socio-technical regime and the socio-technical landscape in a more balanced way. Niche-innovation represents the locus for radical innovation; socio-technical regime includes established industry and current technology and habit of users’ practices; socio-technical landscape encompasses landscape development, (i.e. spatial structures, political ideologies, societal values, beliefs, and macro-economic trends). This approach, should facilitate the identification of desired ESE impacts in the short and medium-term, and the involvement of researchers in the pursuit of those impacts during the project.
Socio-environmental transition is characterized not only by technological changes that in the transport sector, usually categorised as “mature” (in the sense that no radical changes of technological paradigms are expected, compared to sectors like ITCs or biotechnologies) are quite marginal, but specially by changes in users’ practices, definition of new regulations (i.e. land use patterns management), introduction of new mobility services dealing with low cost systems of private and public transport (car sharing, carpooling, urban bike systems…), etc. Differently from environmental transition of transport systems, more oriented to technological changes (i.e. between fuel and electric vehicles), socio-environmental transition is oriented to analyse changes in user practice and lack of regulation where EU TRI should play a key role in transition. Socio-environmental transition is holistically based, and points to the social inclusiveness of transport systems in the light of EU environmental objectives.
Added value of EU scale research funding
EU added value is variously acknowledged to be a concept whose concrete application is unclear (Zuleeg and Mlino, 2011) and relatively undefined (Culture Action Network, 2011). Nevertheless, EU added value in relation to research and innovation is of great interest and importance in seeking to understand the role and impact of the EU in this area.
The TRI-VALUE results highlight that there is high EU added value from the activities within FP7 and that most coordinators attribute this value to aspects such as the “network affect” access to complementary expertise, or to overcoming barriers for uptake of results.
Such an asset is hard to value, but there are clear advantages from having several informal networks with members from different communities (research, industry, consultancies, NGOs) and from different countries.
The assessment of the added value results primarily from the interviews with coordinators for the sample projects, being questioned about how they evaluate the added value of EU research funding in terms of:
• Access to complementary expertise
• Understanding the needs and characteristics of a wide market
• Overcoming specific barriers to the market uptake of research results
• Overcoming lack of funding at national level
• Capitalising on previous EU projects experience and networks
• Availability of alternative sources of funding
In general the results are consistent within each single mode.
“Different cultures, with different levels of expertise... it was not possible to conduct this type of project accounting just with one country… the global perspective would be lost!”
In the aviation sector, all the factors were assessed to be important among the projects surveyed. In particular, access to international complementary expertise and overcoming lack of funding at the national level were important in 80% of the projects surveyed. This shows the critical importance of EU funding in air research and technology development.
“Without European funding this project would not be possible to be carried out.”
In intermodal projects, the assessment of added value indicates a focus on understanding the needs of a wider market, though also with strong views on the access to complementary expertise followed by capitalising on previous EU experience and overcoming the lack of national funding.
“The international consortium work enables to verify and think in broaden terms.”
The main added value, stated by all respondents on the rail projects is the access to complementary expertise. Two thirds of the respondents give high scores to the added value of "understanding a wider market", "overcoming lack of funding" and "capitalising on previous EU project experience".
“Took advantage of the knowledge that has been created in Europe and transfer of this globally”
“Results were given to industry and plans made for more industry participation in the next project and plans made for developments subsequent to the next project including exploitation of follow-on project results”
For road projects, access to complementary expertise is also the key added value of EU research. Capitalising on previous EU project experience obtains the second highest score, while two thirds of respondents also note the added value of "understanding a wider market" and "overcoming lack of funding".
“Helped to open new horizons, invitations to other commercial events and be further recognized in the area.”
For urban projects added value of EU scale research is almost equally spread along all categories, with the exception of availability of alternative funding. Access to complementary expertise and capitalising on previous research are however the two main added value areas pointed out by all the coordinators.
In conclusion, evidence reviewed in TRI-VALUE demonstrates considerable added value from European research, as it contributes to promoting excellent science in transport at the European level.
Firstly, it shows that the international consortium work enables researchers to verify and think in conceptually broadened terms. Secondly, in some areas the project topic was highly international (maritime, air). Significant numbers of projects could not have been undertaken without European funding and in some cases, adequate funding was simply not available at national level.
Whilst added value comes from the collaboration across borders and experiences and skills which contribute to a more comprehensive piece of research, national research funding rules often do not permit funding of international projects. For example, it is clear that the added value in terms of helping the European transport industry to lead at the international level is particularly noticeable in major maritime and air projects with partners such as Korea, China and USA. Furthermore, given the complex and global nature of air-transport, often it is mentioned that only European funding could have supported this kind of research.
Achievement of Program Objectives and Impacts
A critical element in any ex-post assessment of a research programme is to determine whether the objectives stated in the work programme and the expected impacts referring to each specific funded project were achieved. These execution indicators were explored in TRI-VALUE through an independent analysis in the sampled projects (i.e. an assessment by the TRI-VALUE project reviewer), that was later discussed with the coordinator in the telephone interview, which allows an overview of performance in this area. The results from this analysis are outlined in Figure 9.
These results show that on average transport FP7 project objectives reflect the work programme and call objectives, with an average assessment of 4.7 (maximum of 6) and more than 90% of the projects ranking more than 4 (i.e. having a positive grade). In terms of the level of achievement of project objectives and expected impacts the results are somewhat lower, with average grades being 4.5 and 4.2 and with fewer projects above 4, namely 83% for the achievement of objectives and 80% of the expected impacts.
Looking at results per mode of transport and project orientation TRI-VALUE concluded that, with the exception of road transport, there are no major differences between the assessments for policy or technology oriented projects. In short this means that we cannot observe a pattern of project orientation influencing the assessment of the achievement of objectives or expected impacts. As for the influence of modes of transport the average assessment grades vary from 4 to 5 (in a scale of 0 to 6) with the exception of rail projects which seem to score lower grades.
The assessment of the achievement of objectives was checked with coordinators in the telephone interview. Major disagreements in the assessment were not found.
This data suggests a generally very good achievement of programme objectives and a good achievement of expected impacts. These reassuring results do not mean that problems do not exist –roughly 20% of projects still show a negative evaluation on achievement of expected impacts.
In order to improve this performance the European Commission could study the implementation of a more thorough ex-post assessment of projects which could range from a grading of performance by the project officer that could be used to profile participants (and applied in future projects evaluation) or the implementation of quality audits for assessing achievement of objectives and expected impacts inspired by those already performed for project financials.
Another important indicator related to the execution of the FP7 transport research programme regards the involvement of SMEs in research activities. SMEs represent a major share of companies and account for most jobs in Europe. Additionally, SMEs are a source of innovation and they conduct a large amount of R&D, but due to the characteristics of research in the Transport sector, the access of SMEs to innovation is not always easy. This issue was deeply debated within the Market-up project, which concluded that “there seems to be an untapped reservoir of innovative transport SMEs to participate in research activities, and there are numerous studies, such as the Eurogia+ programme, noting the importance of the right combination of large enterprises, SMEs and research institutes, as each type of actor brings valuable features to the projects.” (Market-up Project, 2012)
Accordingly, several rules and operational procedures designed to ensure the programme is relevant to SMEs with the necessary domain knowledge and in-house research capacity were included in FP7. The two most important SME features followed were:
• a target of 15% FP7 funding to go to SMEs in the Cooperation Programme;
• a 75% funding rate for SME participants, compared to 50% for large companies.
While all topics are open and recommend the participation of SMEs, actions oriented towards supporting the participation of SME and fostering their role were launched specifically, along all the three sub-themes of transport program. The 15% objective for SME participation has been reached across all the research activities: SME participation overall is 25%. Moreover, as seen above, SMEs take 21% of the total budget and have high shares in terms of coordination responsibilities.
Other studies have looked more specifically at transport-related SME engagement, notably the FP7 Market-up on Transport Research Market Uptake. Within TRI-VALUE itself, a range of analyses were undertaken, focusing in particular on whether SMEs were identified as end users and users of outputs. This analysis is reported in Deliverable 2.1. In summary, the data suggests a fair degree of SME engagement with, for example, 42% of project coordinators in the on-line survey identifying SMEs as potential users of project results while 33% of them have implemented their project results. This general picture is repeated for the other modes examined, although with slightly lower levels in the road sector.
These results can lead to the question of whether the particular interests and needs of SME were duly considered in the elaboration of the work programs. Such concern has been taken into account with a set of call topics addressing specifically SME participation (see below in the table the targeted SME actions). However, the data gave only limited insight into the scale and nature of SME involvement and little relating to topics such as the effectiveness of measures to encourage SME engagement in the short term. Success of projects targeting the support for SME participation will probably only be visible within subsequent programmes.
Another relevant indicator refers to stakeholders’ engagement beyond the research community. The fact the Commission pushes for appropriate dissemination of project activities has impacts in the call for proposals, in the evaluation and negotiation process and in its project monitoring and has certainly contributed to the significant numbers obtained and the potential success story of dissemination activities. A major question is whether those dissemination actions were also able to reach and engage stakeholders beyond the research community in the debate on scientific issues and of research results. Several initiatives in the field of scientific communication and education, including the involvement, where appropriate, of civil society organisations or networks of such organisations, were launched over the programme.
Efforts have been also made towards targeting future generation of researchers and professionals in transportation. Some of those examples include the projects EDUCAIR and, SKILLRAIL in the fields of assessing educational needs and shaping a new generation of workers. Other projects target specifically actions to attract future careers (i.e. FLY HIGHER) and to raise awareness of potential job opportunities (i.e. TECH-CLINIC SST, PROMARC).
Additionally several initiatives towards the coordination of research efforts (i.e. CAPPADOCIA, CORE-JETFUEL, REACT); capitalisation of results (i.e. SOLUTIONS, CAPITAL, TIDE), knowledge transfer (i.e. TURBLOG-WW, SEAHORSE, CO3) have been launched in practically all the transport sub-themes.
TRI-VALUE Conclusions and Recommendations
Considering the results outlined in the sections above, which were discussed with stakeholders and Commission staff at several occasions throughout the project, TRI-VALUE raised a number of conclusions and recommendations aiming to derive messages from this assessment to build constrictive links to improve Horizon 2020.
In total 13 Conclusions and 8 Recommendations were prepared, which are briefly presented below.
Conclusion 1: The Transport Programme shows a large contribution to the achievement of the FP7 objectives and of the “Cooperation” specific program
TRI-VALUE results estimate that 54% of projects developed in FP7 made a contribution to strengthen the competitiveness of the European industry and more than half of all transport research projects developed in FP7 have contributed towards the achievement of increased efficiency of the whole transport system.
In general, there is good alignment between the work conducted under FP7 and the transport policy objectives. In particular, GHG reduction and safety were key focus areas for transport research, followed by reduction of pollutants and energy efficiency. Across all modes there is a large share of projects (i.e. 75%) contributing directly or indirectly to those objectives.
All in all, more than a quarter of policy related projects and close to one fifth of the technology related projects anticipates their outputs to be used by the European Institutions. This, together with the finding that 60% of all projects deliver policy recommendations, materializes a substantial contribution of FP7 to the European transport policy making process.
Conclusion 2: The FP7 Transport programme reveals a strong European Added Value
The analysis of EU added value has demonstrated that many activities would not have been performed without funding from the European Commission. This is reported to be particularly true for aviation and shipping, which are by nature international, and which would be disadvantaged if they had to rely exclusively on national funding for R&D activities.
A second important piece of evidence is the creation of networks of research across Europe. The access to complementary expertise and “network” effect from FP7 projects is one of the most intangible but important assets generated by the transport research. The TRI-VALUE analysis showed that more than a third of projects have been successful in creating a formal network within the scope of the project, and an overwhelming majority of participants (>86%) continued to cooperate even after completing the project for which they originally formed a consortium. Nearly 75% of the respondents also developed contacts with external organisations as a result of a project. This reflects a great support and considerable success by FP7 in building skills and knowledge networks for transport research across Europe.
A third aspect of relevance on the contribution to a knowledge-based society refers to the engagement of the general public in the debate of scientific issues and research results. Although the research community remains a key player in transport research, the analysis shows that several projects have benefited and have derived part of their success through conveying
Conclusion 3: FP7 Transport research shows a good level of achievement of project objectives and expected impacts set out in the annual work programmes
The TRI-VALUE results showed that, on average, transport FP7 project objectives reflect the objectives of the annual work programme and respective calls. On a project basis, there is a good level of achievement of objectives and expected results, with 83% of the projects scoring more than 4 on the achievement of objectives and 80% for the expected results (On a scale of 1 to 6,).
Conclusion 4: The framework for cross-cutting activities could be further developed
The approach adopted for FP7 was an outstanding improvement compared to the previous FPs, including aspects of programme design in terms of main objectives and sub-themes, specifically covering cross cutting activities.
In addition to the TPT programme, which generally addresses multidisciplinary aspects but only accounted for 3% of the budged, cross-cutting topics were included within the modal components of the annual work programmes. This resulted in a somewhat fragmented landscape for cross-cutting topics, partially due to the underfunding of this dimension in the initial FP7 design stage. The approach to cross-cutting activities (including in particular socio-economic, institutional and decision-making issues within the transport sector, which are of cross-cutting nature), however, lacked an appropriate framework, comparable for example to the Strategic Research Agenda of ETPs for modal research. This made it difficult to understand the logic behind topic selection and to properly develop adequate research and dissemination strategies by researchers.
Moreover, the increased attention to intermodality (e.g. one of the major ambitions of the Transport Policy White Paper is optimising the performance of multimodal logistic chains), which is cross-cutting in nature, shall drive a reflection on whether the budget distribution in FP7 , with only 3% of the budget devoted exclusively to cross-cutting issues, may fall too short. Increasing funding in this category within Horizon 2020 is thus recommended.
Conclusion 5: Procedures for priority setting and Work Programme design are well established contributing to the realisation of ETP Strategic Agendas
Annual work programmes are adopted by the European Commission, with the assistance of the Programme Committee. They set out information on the priorities, objectives and policy relevance of the research topics, which would be implemented through calls for proposals in that specific year. In preparing the work programme, the Commission relies on advice from a wide range of stakeholders, including from the European Technology Platforms (ETPs) and the Transport Advisory Group (TAG). Program Committee, with the representatives of Member States, is then responsible for giving final approval of the work programmes.
European Technology Platforms main contribution is to provide the strategic orientations to the work programme design. Strategic Research Agendas present the approach adopted by each Technology Platform to establish research objectives and development priorities, timeframes and action plans (Roadmaps) that tackle the challenges of the respective industrial sectors. The regularly updated versions of the Strategic Research Agendas are important input to the design of the work programmes.
However, based on the assessment of technology-oriented projects (which are the ones usually aligned with the ETP's SRAs), it can be stated that, in spite of a satisfactory result in terms of the achievement of objectives, the answers by coordinators on the exploitation of results reflect a lack of awareness on the use of project outcomes. For example, 50% of coordinators “don’t know” of SMEs that directly implemented project results and/or recommendations and 49% of are also unware of such use by large companies. Moreover, only two coordinators provided some detail on the estimated market value of projects’ results, and one of them was unable to quantify. This could hamper FP7 attempts to move closer to innovation, development and market implementation with the support of ETPs.
Conclusion 6: Transport research provided substantial contribution to the formulation, implementation and assessment of Community policies
With more than a quarter of policy related projects and close to one fifth of the technology related projects producing outputs to be used by the European Institutions, with the indication that 60% of all projects delivered policy recommendations and with project coordinators estimating that 15 to 30% of projects already produced results that were “used to date” by the EU institutions, it is concluded that FP7 activities have a substantial contribution to the European transport policy making process.
The TRI-VALUE evaluation is clearly indicating that in terms of outputs and dissemination, projects are making an effort to be aligned with and contribute to the policy processes. An analysis of whether this effort is reflected in concrete Community policy outputs is very difficult to perform. The research and innovation process has its own long term time-frames which impacts the effectiveness of an ex-post assessment just after the end of a project and programme. This adds to the long timeframe of the policy formulation processes, which is especially true at European level.
Conclusion 7: Transport research has provided initiatives to engage the public, beyond the research community, in the debate on scientific issues and research results as well as in the field of scientific communication and education
The picture in terms of project participations shows a good balance between industry (being large companies or SMEs) and research (being research bodies or educational institutions). Still, this pattern reflects a certain need to improve the participation of civil society organisations, which may play a very important role both in involving the public beyond the scientific community, but also in improving the contribution to policy formulation. Analysis shows that projects have been successful in conveying and disseminating messages beyond the scientific community, especially towards business organisations.
More than 20 projects were specifically designed to target communication beyond the research community. Specific actions to setup and give continuity to a network of National Contact Points to support and foster the active participation of stakeholders in European research calls and projects have been promoted, with large success in reaching a wide universe of stakeholders and in supporting activities.
Conclusion 8: Transport research shows an adequate adoption of measures for dissemination, knowledge transfer and broader engagement
Dissemination of the results of the work undertaken in FP7 has been strong, through conventional conference papers and peer-reviewed journal publications, but also through electronic means. Websites are seen as particularly important, but there are also signs of increasing use of social media for dissemination purposes.
Several initiatives in the field of scientific communication and education, including the involvement, where appropriate, of civil society organisations or networks of such organisations, were launched during the program. A good number of projects is targeting the future generation of researchers and professionals in the transport sector focusing on coordination of research and knowledge transfer, in practically all the transport sub-themes.
Conclusion 9: SME involvement in Transport research largely exceeds the threshold established for FP7
SMEs in Transport research represent 24.5% of project participants (reaching 28% in CSA projects) accounting for 21% of the EC funding in FP7. These data suggests a good degree of SME engagement, which adds to the fact that 42% of project coordinators in the on-line survey identified SMEs as potential users of project results while 33% of them have implemented project results.
The particular interests and needs of SME were considered in the elaboration of the transport work programs, with a set of call topics addressing specifically the SME participation and /or supporting their activities in the program. Data obtained provided only limited insight into the scale and nature of SME involvement and little information relating to topics such as the effectiveness of measures to encourage SME engagement in the short term, and success of projects targeting the support for SME participation. These elements will eventually be addressed in the next programs.
Conclusion 10: Collaborative research in transport allowed the creation of networks of excellence on transport research
As noted above, more than a third of the projects have been successful in creating a formal network within their scope and an overwhelming majority (>86%) of them continued to cooperate even after completing the project for which they originally formed a consortium. Nearly 75% of the respondents indicate that they have developed contacts with external organisations as a result of the project. These numbers confirm the perception, gathered in stakeholder consultation processes, of the importance of this aspect. Being part of an international consortium offers positive spill-over effects widely recognised but not easily measured.
Conclusion 11: The number of International cooperation actions supported has increased but many challenges remain in this area
A new approach to international collaboration was introduced in FP7 integrating the international dimension into each thematic area of the Cooperation programme. Globally, throughout the program, more than 200 international participants were involved and 118 projects that included international participants were funded. Four joint calls resulted in 12 funded projects.
The evaluation showed that challenges remain in this area. It is often unclear what the added-value of the engagement of international partners is and the extent to which this allows, for example, the establishment of networks at global level similar to those that are being established at European level.
Conclusion 12: Transport research responds to transport societal challenges
The analysis has shown that transport research is responding to transport societal challenges, although in different ways across modes.
Globally Climate Change remains the challenge that attained more attention, but with major variances across modes (meaning that more attention may still be needed in some cases). On the other end, security, for which less than 20% of the projects are reported to have direct impacts, appears to deserve more attention across all modes. However, it is important to note that “Security” is one of the topics covered in a dedicated theme within the FP7 ‘Cooperation’ programme, which may justify lower attention within the ‘transport’ theme.
Improving the links between transport research and societal challenges should be better addressed, however, it is clear that a very good basis exists and substantial work has been undertaken and is now being developed in Horizon 2020.
Conclusion 13: Transport research produced positive impacts on innovative activity
The evaluation showed that projects have been successful in producing positive impacts on innovative activity in Europe. Approximately 20% of projects have led to patents, pointing an intention to exploit research results commercially. Participation in projects also enhances the technological development, notably through development and use of new tools and techniques, the design and testing of models and simulations, the production of prototypes, or demonstrators and pilots.
The evaluation outcomes reinforced the need to establish new mechanisms for looking at how the innovation system is functioning along the Technological Innovation System (TIS), providing a framework to assess how effective transport’s TIS is.
On the basis of the above conclusions, TRI-VALUE proposes the following recommendations:
Recommendation 1 – Maximise the contribution from research projects to transport and other policies and establish guidelines/guidance on how to address policy objectives during the research and innovation cycle
Research and transport policy objectives are typically expressed in general terms. They need to be further clarified, so as to be translated into tangible objectives relevant to the different levels of the research cycle. This can be addressed in Horizon 2020 throughout the work programmes (and particularly within the topics' description), the evaluation of proposals and the implementation of the research project. Any sensible monitoring of objectives would need the development of a simple set of performance indicators for each project, to be subsequently assessed upon project completion. This approach would serve to further integrate key policy objectives within the research activity.
TRI-VALUE observed the high contribution of transport projects in the formulation, implementation and assessment of Community policies. Notwithstanding the relevance of that contribution, there is a need to improve/strengthen feedback channels between transport research and EU policies. An obvious way to do that would be through reinforcement of current internal mechanisms of cooperation within the European Commission's relevant services. Furthermore, it can be considered that, for research activities to be more influential in policy formulation, there is a need to revise the relationship between the two. Rather than expecting that research should "underpin" and facilitate policy implementation, a more ambitious role could be envisioned: exploring and identifying "future challenges", widening the policy options available and better integrating transport within key transitional processes in the European society. As such, besides a source of answers, research could recover its role for identifying the right questions. This could be effectively undertaken by including transport issues in the Foresight processes being developed by the EU, for example in cooperation with the European Forum on Forward Looking Activities (EEFLA)
Recommendation 2 – More emphasis on intermodal issues
In order to better address societal challenges, more attention could be put on intermodal transport.
Although some cross modal issues were addressed in the modal categories, achieving “integrated transport” is a major societal challenge the importance of which does not seem to be properly reflected in the budget allocation for cross-modal and cross-cutting issues. Re-balancing the expenditure to better meet the policy relevance of each aspect is a potential area for improvement in Horizon 2020.
In practice this may imply a redistribution of the budget allocation from a mode-by-mode basis and based on the actual contribution of each mode for European transport and/or for addressing societal challenges. One concrete aspect that should be subjected to debate is whether the mechanisms for share of funding between modes should more directly reflect some of the policy orientation (e.g. share of GHG emissions). A careful analysis is recommended, notably looking at future trends and dynamic aspects, such as industrial competitiveness, knowledge and technology transfer across modes.
Effective support to the integration of transport systems would require increased communication among modal research systems, currently organised through ETPs, mainly with a modal structure. Consistence among modal SRAs should be analysed with view of giving due attention to integration issues. Furthermore, a strengthened focus on social, in addition to purely technological issues within SRAs should be included.
Recommendation 3 – Improve conditions for the uptake of transport research results
Despite the positive results obtained in terms of the production of outputs and exploitation of results, a relatively large number of project coordinators did not appear to know what happens to their project outputs after the end of the contract period. This observation suggests a need to focus more on the uptake of transport research results.
This may require an effort in three directions: (1) better identifying those topics with more potential for exploitation, during the preparation of the work programme by the European Commission; (2) better assessment of the exploitation potential and interests of the research team during the evaluation and selection of proposals, and (3) more precise exploitation plans and actions during the implementation of the research project. Across all these phases, detailed exploitation plans may be needed with concrete targets established in 1-2-5 years (i.e. this could be up to 5 years keeping the same metric as the financial obligations). An improvement in this direction may be observed across FPs, following the high priority that has been attributed to dissemination at all levels. Establishing a feedback loop with reporting mechanism on achievements in 2-3 years after project conclusion should also be considered. In addition dissemination efforts could be re-oriented towards a more selective approach addressing best performance in research and highest exploitation potential. In fact current dissemination approach is equitable but focused on itself and, consequently reported by project coordinators very effective. However, dissemination can be a powerful channel for market uptake if streamlined to achieve that goal. This can be addressed within the changes introduced by H2020
These changes are not straightforward, given the fixed-term nature of research contracts, the fact that individual’s move on to other responsibilities, and that research impacts may take many years to surface. However, this would enable a more realistic assessment of economic indicators. The question of better exploitation and reporting feedback has been consistently referred to along evaluation projects but also in projects dealing with the take-up of research results, which makes it a point where the European Commission should devote some attention on.
Recommendation 4 - Improve measurement of impacts and quantification of benefits
The estimated market value of the products or services developed (economic exploitation of outputs) is not known for the majority of the projects. While this may reflect the fact that some coordinators are not in touch with potential end-users, in the TRI-VALUE analysis nearly no concrete figures were provided, with only two coordinators responding to this question. Regardless of all limitations discussed throughout the document, it is hardly acceptable that projects report on the development of products and services with funding from the European Commission and fail to quantify the market value of such products and services as this is a major element for feasibility of exploitation.
Similarly, improvements are needed in the measurement of impacts of transport research activities. Macroeconomic impacts (i.e. employment, consumer price, regional effects, international trade, and public authority budget), the contribution of each research project to the key environmental priorities at EU level and social impacts of projects were constantly addressed in TRI-VALUE questionnaire and telephone interviews. Nevertheless, current data (in final technical reports) doesn’t allow for a quantification of these impacts and gives little room for validating the coordinators’ assessments. Environmental data is not even part of the final reporting templates. Current data is focused on outcome and, at the limit, results can be assessed but impact assessment requires time duration for maturation of effects.
One mechanism to consider is to use “project audits” inspired by those performed for financial assessment purposes. Such audits could assess both technical compliance aspects (i.e. did the project achieve the objectives and expected impacts, whether deviations are justifiable), verify the data included in final reports and promote a thorough assessment of project longer term impacts and benefits. Moreover, specific performance indicators may be added at the proposal stage to facilitate these ex-post audits. The experience of the Intelligent Energy and Marco Polo programmes where expected impacts are estimated in the proposal phase and progress on achievements is reported annually (i.e. x% reduction in CO2 emissions, x% increase in pax.km or ton.km) may be built upon. As indicated above, the identification of such indicators should ideally be supported by more concrete guidance on the research and transport objectives addressed by the relevant work programmes of Horizon 2020.
In addition ex-post assessment should be also performed at later stages, when impacts can effectively be measured.
Recommendation 5 - Establish a Framework to assess the effectiveness of a Technological Innovation System
The scientific community is invited to improve the framework to assess the innovation systems. The Technological innovation System (TIS) approach has been developed to address the actual limitation on assessing how the innovation system is functioning and how effective a TIS is. R&D policy can usefully address weaknesses in these functions. TIS approach is described in Bergek et al. (2008a, 2008b) and has been used by JRC in the mapping of EU transport innovation and in the FP7 Market-up project. They introduce the idea of seven “functions” of an innovation system:
1. Knowledge creation: how the technology can be applied to develop marketable products, performance of the technology, new production processes
2. Guidance of search: what external factors promote or inhibit innovation and which aspects – e.g. performance, safety, environmental impact are externally imposed on innovators?
3. Entrepreneurial experimentation: innovation activities in firms to develop the new technologies.
4. Resource (financial, human) mobilisation: application of investment capital and humand resources in innovation and product development.
5. Legitimation: acceptance in society, adoption of suitable legislation and standards.
6. Market formation (strength of demand): development of new markets or presence of the new technology products in existing markets.
7. Development of externalities or knowledge diffusion through networks: the strength of sharing the knowledge that has been created in innovation networks.
The impact measurement mechanism presented earlier may consider elements of this innovation system
Recommendation 6 – Create a platform to discuss and monitor the competitiveness dimension of the transport programme
The increasing EU focus on the global competitiveness of its transport industry should be translated into a consistent and transparent strategy. On the one hand, competitiveness should be put at the service of a global sustainable transport paradigm that is not yet explicit enough in EU policy documents. On the other hand, global competitiveness should be defined with a consistent path, in order to provide guidance on how concrete research topics and projects may contribute to European competitiveness, rather than to protect the technological advantages of particular players within Europe. Accordingly, it would be worthwhile to provide an adequate discussion platform to discuss and monitor the competitiveness dimension of the transport programme in Horizon 2020. This approach would require improved monitoring of exploitation activities and use of results from research projects compared to those available in FP7. As this did not seem to be a priority for many project coordinators, the implementation of this monitoring should be encouraged by the EC services at least during the first years of Horizon 2020.
Recommendation 7 – Maintain the focus of international cooperation projects on network promotions
Typically, international engagement in projects has been secured by organisations joining as experts rather than as consortium partners. In some cases there is some disappointment with the participation of non-European partners.
Lack of travel budget, associated paper work and short-term projects were some of the reasons mentioned as constraints to international transport collaboration, during TRI-VALUE project interviews and these are all matters that future programmes could, and should, address.
In general, the focus of the international dimension of the FP7 should increasingly be on engaging with partners from countries outside Europe on equal terms and in programmes and activities of high mutual interest focusing on the creation of excellence networks, trying to replicate what European projects have already managed to achieve at EU level.
Additionally, formal transport knowledge networks of excellence in EU could serve as an effective means to better target EU resources and to establish a consistent collaboration policy at the international (global) level. A "transport research agenda for leadership" could serve as a reference for stakeholders and as a valuable means to bridge the gap between transport policy at the European and global level.
Recommendation 8 - Increase engagement of Civil Society organisations
The analysis performed by TRI-VALUE shows that projects have generally been successful in conveying and disseminating messages beyond the scientific community, especially towards business organisations. It also shows that while the programmes remain somehow oriented towards promoting their outputs in the research community, engagement of business organisations, including SMEs has been quite successful, which seems to help, at least in some cases, improve the commercial exploitation of results.
Horizon 2020 should build from the progress on improving involvement of businesses in TRI, which has been a focus in recent years, and put in place a strategy to improve the involvement of civil society organisations, that is undertaking knowledge transfer near the citizen. This is expected to be an effective instrument to disseminate results to a larger audience, well beyond the research community, and to also bring benefits to improve links between research activities and policy formulation, as many civil society organisations are active stakeholders in policy processes. As a first step in increasing participation of civil society organisations a review of their interests and barriers for participation may be of relevance. Accordingly we propose that the strategies to enhance participation of large companies and SMEs are critically reviewed, with the view of identifying potential ideas for further engaging civil society. For example, the advantages and disadvantages of increased involvement in ETPs or of a “target” for participation of civil society organisations in the programme shall be discussed.
Another step to achieve this may be to give adequate relevance to social issues within transport research projects, which would provide an excellent basis for the involvement of civil society organisations, and for the development of better tailored dissemination activities by researchers. To a certain extent this is already underway, as Horizon 2020 aims to focus particularly on addressing societal challenges.
Furthermore, a more fluid relationship between research services and transport services is necessary to adequately identify topics that deserve research funding; otherwise, there is a mutual risk for either choosing a topic by its innovative character without much regard for policy priorities, or choosing projects with limited- if any- innovative character, as a sort of technical assistance for too narrow, short-term policy issues. While collaboration already exists, notably at the stage of preparation of work programmes, it is recommended to ensure that such collaboration within the Commission services and with the outside world is extended to cover the entire project lifecycle. This step forward is necessary for transport research to remain relevant and to provide added value at the EU level to help formulate the right questions and to stimulate and sustain fruitful and transparent public discussion within Europe. As noted in recommendation 1, this could be organised through linking with Foresight activities of the EU.
Potential Impact:
This section looks at the overall impacts of TRI-VALUE and to dissemination and exploitation of project results, which greatly influences the extent to which the expected impacts are achieved. It is structured in two parts: the first looks at the TRI-VALUE impacts while the second addresses the measures for dissemination and exploitation of results.
Impact
The objective of TRI-Value was to undertake an ex-post evaluation of the Transport Theme within the FP7 programme. This included the assessment of the implementation and management of the FP7 transport programme, as well as the identification of the achievements and impacts of research transport projects co-funded within FP7. An integrated programme or project evaluation should include the specific objectives, its economic, social and environmental impacts, and its efficiency, effectiveness and relevance, taking as a reference the role that transport research is expected to play within the EU's relevant policies during this period (2006-2013) and beyond.
Therefore, the global and sectoral EU policies constitute the basic reference for assessing the impact of the FP7 programme, reflecting the attention within Europe to integrate crucial objectives in an increasingly challenging context, progressively dominated by the global financial and economic crisis:
- Reducing the environmental impacts of mobility, and particularly GHG emissions, through the identification and implementation of innovative technological solutions and policy measures.
- Supporting the EU transport industry and its global competitiveness.
- Addressing the emerging mobility needs linked to the six “grand challenges” faced by European societies.
TRI-VALUE fully integrated this background within the evaluation framework at both the programme evaluation level and the project evaluation level. The evaluation provided insights on how the revision and fine-tuning of EU policies in the period (2006-2013) has influenced the FP7 transport programme, and how projects have reacted to the increasing attention within topic descriptions to the emerging global environmental, societal and economic challenges. By doing so, TRI-VALUE provided information on how transport research reacts to strategic changes in European policy, and through which mechanisms, which can be particularly useful for the future design and deployment of Horizon 2020.
Furthermore, the influence of EU policy on the contents of the transport research programme and on its projects' results operate through the action of the various agents active in the R&I, including individual researchers and research institutions, the wider R&I community (including the users of transport research results, and most particularly the EU transport industry) and the European, national, and local transport policy-makers and decision-takers (European policy community). TRI-Value explored, through the review at the programme and project levels, the evolving roles of these key actors, and made an effort to actively include them in the evaluation process through a variety of participation and dissemination tools.
As noted at proposal stage, the analysis of the impacts of TRI-VALUE is a difficult topic in itself. Each of the project objectives was articulated at a relatively high level of generality. Collectively, they constitute the overall sense of direction that European transport research has sought in recent years. However, they are difficult concepts against which to gather evidence of the effectiveness of individual research projects, or, indeed, even research programmes.
In the next paragraphs the impacts of TRI-VALUE are presented with greater level of detail according to the target group: the community of researchers and organizations concerned, the community of transport researchers, the Industry, the Transport Community and the Final Users, and the European Policy community. In the end a specific summary section outlines the most important impacts in terms of efficiency, effectiveness and relevance of TRI-VALUE project activities.
Impact within the community of researchers and organizations concerned with the methodology of project and programme evaluation
A carefully conceived, clearly articulated and competently collated evidence base is critical for an effective and broad impact of any ex-post evaluation exercise. Considering the limited time and resources available to undertake the ex-post assessment, TRI-VALUE project decided not to develop a new evaluation methodology, employing instead the Research Impact Pathway (RIP) model of evaluation, with some fine-tuning activities.
Once again the RIP has proved effective in providing a structure and rationale to guide the assessment, which can be communicated and understood by users and project stakeholders. The review of other research assessments performed in an initial stage of the TRI-VALUE project proved to be a useful exercise, notably supporting the introduction of the auscultation of end-users in the assessment process.
A good evaluation methodology and practice is important to many government and quasi-governmental organizations. The framework applied in TRI-VALUE is in itself a contribution to the improvement of management practices in these organizations. Moreover, several conclusions and recommendations address the need to improve the mechanisms to measure research impacts. If properly followed-up by the European Commission and other research funders they will allow for an improvement of the capabilities to measure the outputs and impacts of research programmes. Accordingly, there may be an impact on the international community of evaluation researchers and practitioners who may benefit from the positive and negative experiences and insights of the assessment performed in TRI-VALUE.
Impact within the community of transport researchers
In the proposal stage this was identified as the level at which TRI-Value would be expected to have its largest direct impact. Social progress in areas such as transport comes from the interaction of two ‘blades’ of a pair of scissors: one ‘blade’ consists of the policy objectives articulated by elected representatives (subsequently fleshed out and developed by public officials working in national and international organisations) and the other ‘blade’ consists of the knowledge that resides within the research community (about the how and the what of effective transport research). TRI-Value aimed to communicate to the international community of transport researchers on how and what impact they were having with policy makers.
A good fit between the research approach and the research problem is a major determinant of a successful outcome. With its activities, including those related with the data collection process, TRI-Value promoted a large scale reflection on the ways research was organised and its wider impacts. While it was concluded that these links are very hard to establish TRI-VALUE believes that the process of reflection performed in the framework of the project – which involved hundreds of members of the transport research community, from the researchers themselves to end-users or Commission project officers – does have an impact of putting the issue of the impacts of research in the agenda. Moreover, the results of the TRI-VALUE analysis do throw light on how to undertake research tasks in an effective way.
The TRI-VALUE review of FP7 research efforts also allowed identifying remaining and emerging knowledge gaps, which should be addressed in the future. This will have a significant impact on the transport research community, in terms of how it positions itself and develops its skill base to address emerging challenges.
Impact within the Industry, the Transport Community and the Final Users
Although a part of the transport research community discussed above, the transport industry has a particularly relevant role within the innovation cycle, and has played an increasing role within FP7. The transport industry plays different roles within transport research: as a major user of innovation, and responsible for the crucial process of pushing innovation towards market implementation, the industry has played an increasingly influential role in setting the FP7 transport research agenda, mainly through the activities of the ETPs. Furthermore, the industry has been actively involved as partner in an increasing number of research projects. Last but not least, the industry is a key target of the dissemination activities of most of the projects funded by FP7.
TRI-Value reviewed the role of the industry in FP7 as an integrated dimension of its evaluation process. This review highlighted weaknesses and strengths of the industry participation in transport FP7 research and discussed how it has contributed to the global competitiveness goals stated within the "Innovation Union" flagship initiative. These results, which were disseminated at Europe’s flagship transport research event will hopefully raise awareness on the FP7 results among EU industrial stakeholders and will serve them to better tune their strategies and approaches to Horizon 2020.
The transport community includes also operators and managers of transport infrastructure and services and their final users, as well as a variety of social and economic actors. For them, FP7 projects should have been effective in providing new evidence for a better understanding of the transport system, its complex socio-economic impacts and its influence on their respective areas of interest. Within the review process, TRI-Value assessed how these stakeholders have been reached out by project and programme activities, and provided targeted recommendations to further empower these stakeholders and to raise their interest and awareness on FP7 research results.
Impact within the European Policy community
At the broadest level, the key impact of TRI-VALUE is to underpin the effort to keep improving the role of transport research for achieving the vision of the sensitive European transport system. As the Framework Programme moves forward towards the new Horizon 2020 concept, the FP7 final review proposed by TRI-Value was able to provide a picture on how research interacts with transport policy makers and decision takers at the European, national, regional and local level. This analysis also allowed the identification of relevant areas which were less covered in FP7 and deserve further attention in Horizon 2020.
It was clear from the onset that TRI-Value will be just one component of a much broader debate at Commission and other levels. However, the analysis performed in TRI-VALUE was debated with people from different Commission departments responsible for transport research, making the project a forum for an evidence driven debate not only on the results of past transport research but also on how to improve them in the future.
Summary of TRI-VALUE Impacts
As noted above, TRI-VALUE delivered a better understanding of the functioning and impacts of transport research activities financed within the FP7. A set of four areas were assessed: implementation and management of the programme; achievements and impacts of transport research; efficiency, effectiveness and relevance of the funding; and sustainability and utility of different research programmes. In each of these areas the project produced tangible results, all contributing to the final aim of assessing how the EU’s investment in FP7 transport research has contributed to greener smarter and more integrated transport systems and to the achievement of policy goals (Europe 2020, Innovation Union and Transport Policy). We present below a summary of the impacts that each of those results are expected to generate.
Assessment of the level of achievement of programme implementation objectives, users’ satisfaction with management procedures and comparison of the performance with programmes in other economies
TRI-VALUE analysis clearly demonstrated the level of achievement of programme objectives and provided insights on how to ensure that the performance is measured and improved. TRI-VALUE work in this also provided information on how research interacts with transport policy makers and decision takers at the European, national, regional and local level. Concrete recommendations for improvement of programme management procedures were provided.
TRI-Value extended the current understanding of the theory and practice of project and programme evaluation in transport, providing an innovative methodology which allows an effective combination of quantitative and qualitative assessment
The work of TRI-VALUE clearly increased the knowledge-base on the assessment of research and innovation funding programmes. This allowed for the introduction of several improvements in the Research Impact Pathway approach and data collection mechanisms that allows both quantitative and qualitative assessment of research funding programmes to be undertaken simultaneously.
TRI-VALUE evaluated FP7 transport research projects and assessed the extent to which they achieved the specific objectives listed in the work programme and respective calls; the project also assessed the economic, social and environmental impacts of transport research co-financed by FP7
The analysis performed in TRI-VALUE allows for the identification of patterns of success, i.e. a better understanding of how to undertake research tasks in an effective way. Moreover the project identified remaining and emerging knowledge gaps, which should be addressed in the future. The project provided concrete results of research funding and assessed positive and negative outcomes as well as a qualitative and quantitative assessment of impacts on excellent science, industrial leadership and societal challenges.
Measurement of the achievement of specific objectives listed in the work programme or calls and of the contribution to achieve major EU policies, taking into account the investment in transport research and innovation; Assessment the value added of research at EU level and comparison with other regional, national or international innovation policy landscapes in large and countries
TRI-VALUE provided a picture on how research interacts with transport policy makers and decision takers at the European, national, regional and local level and promoted a better understanding on how research results have influenced decision takers and policy makers in formulating their transport visions.
Results also raise attention to areas of new concern towards which future transport research efforts could be well advised to focus attention.
TRI-VALUE outlined patterns of collaboration with research programmes in other economies and provides concrete recommendations for improvement of transport research funding programmes. As a result it is an effort to keep improving the role of transport research for achieving the vision of the sensitive European transport system proposed by the EU policy documents.
Dissemination and Exploitation of Results
As TRI-Value is an ex-post evaluation type of project, and therefore it is focused mainly on activities related to data collection, assessment and analysis of results and development of conclusions and recommendations, the dissemination strategy has been based on the activities that: (i) efficiently engages participation of relevant stakeholders to the project activities (i.e. data collection consultation meetings and events), (ii) creates awareness about the objectives of the project and the upcoming activities, and (iii) informs interested parties about the main findings and outcomes of the project.
Therefore, the main dissemination activities implemented during the TRI-Value project were the following:
- Identification, profiling and engagement of stakeholders (i.e. ETP - European Technological Platforms, representatives of the EC (Directorates-General), Participants from FP7 projects (coordinators and partners) and Other stakeholders (Universities, industry, etc.) through direct contact, or intermediaries (e.g. ETP secretariats, PO);
- Attendance to Civitas Forum and awareness concerning the realisation of questionnaire to coordinators
- Presence in TRIP Portal – dissemination of project´s objectives and activities (e.g. questionnaires);
- Presence in ETP websites;
- Creation of LinkedIn group, for broad dissemination of project activities, results and announcements on relevant events, etc.;
- Participation in a session of TRA2014 (Paris) to present the main findings of the project.
- Presence in the Inforday for Horizon 2020
- Organisation of 2 Workshop with EC and relevant stakeholders to present the main findings and gather feedback;
- Production of relevant visual identity (logo, PPT and word templates) to be used when necessary.
The exploitation of TRI-VALUE results will be fourfold.
First, the European Commission services are expected to use the information made available by the project as an input to a wider ex-post assessment of the FP7, thus helping the Commission to comply with its legal obligations towards the European Parliament and Council. Second, the European Commission services are also expected to make use of the conclusions and recommendations of TRI-VALUE to improve transport research and innovation in the new framework programme, i.e. Horizon 2020. Third, the research community, in particular the ETPs, has in TRI-VALUE a basis for reflecting on their participation in transport research programme, and thus can use the results of TRI-VALUE as a starting point for identifying areas for improvement and selecting appropriate measures to be undertaken. Fourth, the research community as a whole, but in particular consortium members, may find in TRI-VALUE a methodology for assessment of research instruments which may be transferable to other areas. Academic-oriented partners may use this information for publications and/or teaching materials and consultancies may exploit this methodology to position themselves as research programmes assessment service providers, which may bring additional business to the participant consultancies (in particular the SMEs).
List of Websites:
Project Website (LinkedIn Group): http://goo.gl/Tbn4kg
Coordinator Contact Details:
Ms. Daniela Carvalho, daniela.carvalho@tis.pt
TRANSPORTES, INOVAÇÃO E SISTEMAS, S.A.
AV. MARQUÊS DE TOMAR, 35, 6ºDRT, 1050-153 LISBOA, PORTUGAL
T: +351 213 504 400
www.tis.pt
TRI-VALUE looks back into the work of FP7, Europe’s most important research funding instrument, on transport with the aim of per-forming an ex-post evaluation. The overall aims of TRI-VALUE are grouped into three pillars which are then linked to a set of five specific objectives:
• Assess pillar: (1) Measure concrete results of transport research funding in Europe, assessing positive and negative outcomes of research activities analyse implementation and management
• Evaluate pillar: (2) Evaluate quantitatively or qualitatively economic, social and environmental impacts of Transport Research and Innovation (TRI); (3) Measure the EU added value of transport research funding, notably assessing its contribution to foster excellent science, industrial leadership and to address the societal challenges; and (4) Measure the relation of the scientific and technical objectives in TRI programmes with the achievements in major EU policies, such as the Europe 2020 strategy, the Innovation Union Flagship Initiative and the ‘grand challenges’, and the White Paper on transport policy; seek to identify areas where further attention appears to be needed.
• Compare pillar: (5) Compare the performance of EU’s TRI with the approach in other major economies.
This information will be used to outline conclusions and recommendations for improving transport research and innovation.
The ex-post evaluation was evidence driven, i.e. a set of qualitative and quantitative information about the research and innovation work in FP7 was collected and critically analysed. This allowed the team to combine several perspectives on a single project, which improves the reliability of results and may correct eventual bias (e.g. by the coordinator who may be optimistic or by a reviewer with little contact with the project). The assessment was based on the application of the Research Impact Pathway (RIP), a methodology that helps to overcome some of the major limitations associated with these types of assessments, notably the fact sponsors of research want to assert impact soon after the research is undertaken whereas, in reality, many impacts will only develop gradually over (many) years.
The review performed in TRI-VALUE provides overwhelming evidence that research and innovation projects are producing a large quantity of relevant outputs. However, only a small share of projects delivered new services, new products, new norms and standards or patents, which may reflect the fact that TRI in FP7 is still focusing much more on basic research and early stage development rather than closer to market applications. It is still too early to say whether the project results will result in economic exploitation in many cases, however when looking at how the outputs are used and adopted by the relevant communities, results are less visible than the overall magnitude of the outputs production, with the research community emerging as the main user of the outputs produced. While most FP7 transport projects might have wider societal impacts in some way or another, in most cases those impacts will only arise in the long term and were not possible to quantify in TRI-VALUE. This a need to improve the measurement and quantification of project benefits and impacts and a new mechanism for following the impacts of projects after the official completion date (i.e. 5 years). In general, there is a positive evaluation of projects’ consideration of policy objectives in their work, in particular for GHG reduction and safety, and TRI-VALUE analysis has shown that transport research is responding to transport societal challenges, although in different ways across modes. There is still room for improvement regarding the links between transport research and innovation and the societal challenges; however, it is clear that there is a very good basis since there is already substantial relevant work taking place. TRI-VALUE also assessed the extent to which transport research projects are contributing to policy formulation. Results show that this is a major component of transport research and innovation.
TRI-VALUE evaluation concluded that the Transport Programme shows a large contribution to the achievement of the FP7 objectives and of the “Cooperation” specific program and reveals a strong European Added Value. FP7 Transport research also shows a good level of achievement of project objectives and expected impacts set out in the annual work programmes. Although multidisciplinarity was encouraged throughout the Transport Program, cross-cutting activities still lack an appropriate framework. Procedures for priority setting and Work Programme design are well established contributing to the realisation of ETP Strategic Agendas. Transport research had a substantial contribution to the formulation, implementation and assessment of Community policies and has provided initiatives to engage the public beyond the research community, in the debate on scientific issues and research results as well as in the field of scientific communication and education. This reflects the fact that transport research shows an adequate adoption of measures for dissemination, knowledge transfer and broader engagement. Regarding SME involvement, Transport research largely exceeds the threshold established for FP7. The collaborative research in transport allowed the creation of networks of excellence on transport research but, while the number of International cooperation actions supported has increased this cooperation is still limited. TRI-VALUE also concluded that transport research responds to transport societal challenges, although in different ways, and it produced positive impacts on innovative activity.
On the basis of the above conclusions, TRI-VALUE proposed the following recommendations:
• Improve coherence between research and transport policy objectives and establish guidelines/guidance on how to address them during the research and innovation cycle
• More emphasis on cross cutting issues
• Create favourable conditions for the uptake of transport research results
• Improve measurement of impacts and quantification of benefits
• Establish a Framework to assess the effectiveness of a Technological Innovation System
• Establishing consistent global strategies, addressing sustainability and competitiveness
• Focus international cooperation projects on network promotions
• Increase engagement of Civil Society organisations in research activities
Project Context and Objectives:
TRI-VALUE project performs an ex-post evaluation of transport research activities in Europe’s most important instrument for financing research, the Seventh Framework Programme of the European Community for research, technological development and demonstration activities (FP7). In addition to the ‘look back exercise’, TRI-VALUE ex-post assessment aims to learn from this experience and draw lessons for future European activities in this field. Accordingly, efforts are also placed at analysing how the experience of Transport Research and Innovation (TRI) in FP7 may be used to improve the next framework programme – the Horizon 2020.
Faced with an unprecedented economic crisis and increased global competition, the European Union (EU) has defined a strategy to support growth and job creation - Europe 2020 – aggregating seven flagship initiatives and in which boosting the knowledge economy and innovation is a key pillar. Accordingly, the ‘Innovation Union Flagship Initiative’, which aims to improve framework conditions and access to finance for research and innovation so as to ensure that innovative ideas can be turned into products and services that create growth and jobs, tops the list of such flagship initiatives. One of the key elements of the ‘Innovation Union Flagship Initiative’ was the setup of a new research and innovation programme for 2014-2020, ‘Horizon 2020’, which succeeds FP7. Horizon 2020 builds from the experience of previous programmes, thus reinforcing the importance of ex-post assessments such as the one performed in TRI-VALUE.
According to the Europe 2020 strategy the ‘Innovation Union Flagship Initiative’ will aim to re-focus R&D and innovation policy on the challenges facing our society, such as climate change, energy and resource efficiency, health and demographic change. This is particularly important for the transport sector, whose challenges summarized in the Transport White Paper touch upon all these areas. As stated in the White Paper, looking 40 years ahead, it is clear that transport cannot develop along the same path and to implement its new strategy innovation is essential. The six ‘Grand Challenges’ on which R&D will focus in the next programme can be summarized as:
- Global warming: In the current changing climate, how to better manage (prevention and recovery) the consequence of natural disasters such as flooding, fire forest, hurricane, dry area extension which tend to increase in numbers but also in intensity ?
- Tightening supplies of energy, water and food: In a constrained resource environment, how to improve the efficiency of consumption, the recycling rate while further reducing waste ?
- Ageing societies: As the life span of people increases, this raises numerous issues among which are economics, social inclusion, accessibility.
- Public health: How to provide medical care to everyone while minimising discrimination ?
- Pandemics: With the global and fast circulation of people and animals, disease can spread fast. How to improve the prevention and recovery in case of wide spread diseases since they potentially have a huge impact on the economy but also social stability.
- Security: How to improve the security of European citizens and their goods within but also outside Europe?
In summary, both European Transport policy and the Research and Innovation policy are facing transformational reforms. By looking back at the work on transport research and innovation in recent years TRI-VALUE will provide indications for shaping these reforms.
Horizon 2020 also introduces important changes to the research and innovation framework in Europe. Amongst others there is the commitment to focus resources on three priorities: Excellent Science, Industrial Leadership and Societal Challenges. Accordingly, TRI-VALUE will aim to help the European Commission in using the experience from FP7 to improve the way transport research is funded at the European level and in tailoring it to these new priorities.
The TRI-VALUE project objectives have as a starting point a project concept based on three pillars, schematically represented in Figure 1, and which will all contribute towards a main objective of providing recommendations for improving TRI framework in Europe.
The general objective of TRI-VALUE was further detailed, taking into account the project three pillars, for which specific objectives were identified.
Within the “Assess” pillar activities focus on the concrete measurement of the performance of TRI in the FP7 programme. The specific objective of this pillar is:
1. Measure concrete results of transport research funding in Europe, assessing positive and negative outcomes of research activities.
In the “Evaluate” pillar the data collected and the results of the assessment phase are further explored in order to look at actual impacts of TRI. It has the following specific objectives:
2. Evaluate quantitatively or qualitatively economic, social and environmental impacts of TRI;
3. Measure the EU added value of transport research funding, notably assessing its contribution to foster excellent science, industrial leadership and to address the societal challenges; and
4. Measure the relation of the scientific and technical objectives in TRI programmes to the achievements in major EU policies, such as the Europe 2020 strategy, the Innovation Union Flagship Initiative and the ‘grand challenges’, and the White Paper on transport policy; seek to identify areas where further attention appears to be needed.
The third pillar, “compare”, further extends this process of expanding the scope of the analysis, by focusing on the comparison of the performance in European TRI with the work developed in other economies. The specific objective of the “compare” pillar is the following:
5. Compare the performance of the EU’s TRI with the approach in other major economies.
In addition to these five specific objectives that may be directly related to the three pillars defined in the TRI-VALUE project approach there are two other specific objectives with a horizontal basis:
6. To identify a set of indicators to assess (1) the implementation and management, (2) the achievements and concrete results, (3) the economic, social and environmental impacts of transport research programmes and (4) the efficiency, effectiveness and relevance of funding; and
7. Provide targeted recommendations and conclusions for potential improvements in Europe’s TRI and in its practical implementation.
The ex-post evaluation performed in TRI-VALUE is evidence driven, i.e. a set of qualitative and quantitative information about the research and innovation work in FP7 were collected and critically analysed to attain the objectives stated above. The data and evidence base has been drawn from the data collection activities of the Project:
• Desk Reviews of the documentation for a sample of projects;
• Telephone interviews with a sample of Project Coordinators
• Online questionnaire responses from a sample of Project Coordinators;
• Face-to-face discussions with stakeholders at workshops and interviews.
The results collected through these various processes allow for several perspectives on a single project to be considered, from quantitative data collected from EC databases to the coordinators’ assessment or an independent evaluation by an independent project reviewer. This largely improves the reliability of results and may correct eventual bias (e.g. by the coordinator who may be optimistic or by a reviewer with little contact with the project).
Then a methodology to guide the assessment was used - The Research Impact Pathway (RIP) - first conceptualised for SITPRO (FP4 (Transport) evaluation) and further developed in the SITPROplus (for FP5-FP6 (Transport) evaluation). The thinking behind the RIP is that many of the impacts of a research programme have yet to materialise and that impacts will occur over a long time span, transcending the lifetime of individual research projects. The RIP provides a four-element sequence (production of outputs > dissemination > exploitation > impacts) for understanding how impacts can be expected to occur. Observations can then be made relative to this expectation for each individual project.
A key feature of the RIP is to record concrete evidence of progress along the Pathway and to avoid (sometimes wishful) speculation about what might happen. This may help to overcome some of the major limitations associated with the assessment, notably the fact sponsors of research want to assert impact soon after the research is undertaken whereas, in reality, many impacts will only develop gradually over (many) years. In this case this limitation was even more relevant, since although this exercise refers to an ex-post evaluation, from the 623 projects funded by the Transport program, only 336 projects (54% of the total) have been concluded at the start of the current evaluation. For the remaining projects, 110 (18%) were still ongoing and 177 (28% of the total projects) had their kick-off of activities after September 2012, date established as “cut-off” for early starters (i.e. projects with less than one year duration for which the assessment of outputs and results is hardly measurable).
Project Results:
TRI-VALUE project results were organised in two layers, the first corresponding to the production of programme level indicators, which focus on statistics addressing implementation of research activities and allow for an overview of progress in the development of transport research activities in FP7; and the second layer refers to the project level analysis, in which a detailed assessment of the results and impacts of projects is provided. Following the presentation of the main TRI-VALUE results for programme and project levels, the project conclusions and recommendations are described.
Programme Level Results
TRI-VALUE findings at the programme level, focus on the research implementation statistics: time to grant, budget breakdown, participation numbers and breakdown per type of participant, success rates and frequent collaborations. This data was exclusively obtained from the analysis of the EC database CORDA.
Time to grant
Time to grant, i.e. the time lag between proposal submission and start of the contract is a measure providing an assessment whether administrative simplification measures have been effective. A considerable reduction in the number of days to grant during the programme can be noticed: from the first to the last call in FP7, the time was reduced by 151 days (almost 3 months). To achieve this, the different simplification processes adopted, notably the electronic submission of proposals and negotiation files, were certainly large contributors.
An increase from 2007 to 2008 was noticed. This can be partly explained by the lack of a call in 2009, leading to some of the last calls in 2008 being delayed to start not until 2009.
Looking by sub-theme of the Transport program one can note that Surface Transport takes more time to grant than the other programs, in particular if compared with Aeronautics, where project size in terms of number of participants is in general comparable.
Figure 2 outlines the results regarding time to grant of FP7 transport research projects.
The FP7 transport budget
Total EC funding in transport calls over the FP7 period reached the 2.223 Mio €, of which:
• 52% Surface transport
• 45% Aviation (i.e. Aeronautics and Air Transport)
• 3% Cross cutting (horizontal activities)
About 75% of the EC funding in FP7 was allocated to four main objectives: Transport Greening, Efficiency of Transport System, Competitiveness and Safety & Security, present in the work program objectives. Modal shift and decongestion and urban mobility represented 7% and 8% respectively. Pioneering of transport took 5% of the budget while Policy Support took 6%.
This data supports the observation already made in the Intermediate Assessment that Policy Support and cross cutting issues were less covered in FP7, where there was a clear predominance of a modal basis.
Looking at the breakdown of FP7 transport average budget per project, it is possible to observe that budget for CPs (4.9 M€) is significantly higher than for CSA (1.1 M€), given the scope of activities under each of the funding schemes. Per sub-theme, it can be noted that budget per project in aviation (5.7 M€) is, on average, almost 1 million more expensive than the ones in SST (4.6 M€), for the same CPs.
In terms of budget breakdown per type of entity, more than 50% of the total EC funding (54%) is allocated to private sector participants (33% private companies and 21% SME). Research Organisations and Universities each get approximately 19% of the EC funding. Public Body participants take 5% of the budget.
Participations in FP7
During FP7, 2 679 proposals have been submitted corresponding to 29 584 applicants. Of those proposals, 623 were funded as projects, involving in total 8 302 participants. Table 1 presents the detailed breakdown of the number of proposals and projects per type of project and sub-theme.
Collaborative projects represent the majority of FP7 transport projects. 73% of the projects funded are CP and only 27% are Coordination and Support Actions. Since the FP7 programme was particularly targeted to address scientific and technological research, development and demonstration, this result is not unexpected. Cross cutting issues and horizontal activities, where the large majority of the CSA fall, represent a minor share compared with AAT and SST.
Figure 3 depicts the average number of participants per project and the type of participants, allowing for a more detailed analysis of the participations in transport FP7 activities.
It can be observed that, as might be expected, faced with wider scopes and budgets, the average number of participants is higher in CP than in CSA for both Aviation and Surface Transport. In the cross-cutting projects the average number of participants is almost equivalent between the two types of funding. For the type of participants it is particularly interesting to note the 24.5% share of participation by SMEs in FP7 transport projects, compared with a target of 15%.
Success Rates in Transport research
The success rate in the Transport FP7 programme is overall around 23%, reaching 30% for CSA but only 21.5% for CP projects. In all the sub-themes the success rates for CSA are higher than for CP; only for TPT (cross cutting activities) does the success rate presents a minor difference between the different funding schemes.
Regarding success rates per type of participants, the highest can be observed for Private Companies (34%), followed by Public bodies with 31%. Research Centres and Other show success rates around 30%. The average success rates for SMEs are 25% while for Universities it is 24% and 16.6% for Research Organisations. Success is higher in CSA than in CP, however it is noted that Private Companies have a high success rate (32.9%) in CP, compared to the other entities.
The same analysis performed only for the coordinating entities shows again that Private Companies are more successful in their proposal submission as coordinators (in average 31%). The success rate for Research Organisations as coordinators is nearly 25%, and is around 26% for public bodies.
Most frequent collaborations (Research Networks)
Figure 4 highlight the most frequent collaborations between Member States in the funded projects (ignoring collaborations within the same country). Overall, the strongest relations can be found among France-Germany-Italy-UK-Spain as the main players with the strongest pair being France-Germany.
Looking at each of the subprogrammes, i.e. aviation, surface transport and cross cutting, one can see some interesting differences. First, on aviation one can note a great project concentration in Germany and France, which happen to be the two most important Member States for EADS/Airbus. As for cross-cutting issues and SST Italy also emerges as a key player, sidelining with Germany and France.
Program Assessment by Project Officers (based on RESPIR)
TRI-VALUE also considered the assessment of respective Project Officers. Results highlight an extremely positive evaluation of the final results in the Transport projects in FP7, with just 2 in 227 cases, failing to achieve objectives and /or partly achieving objectives obliging to corrective actions.
54% of the 227 completed (and processed) projects fully achieved its objectives and 45% has achieved most of its objectives. Although 98% of the concluded projects available from RESPIR (227) fully accomplished and/or achieved most of its objectives, the assessment of outputs doesn’t show such a positive picture. Indeed, just about one fifth of the projects were identified by the respective PO as a “success case story” and producing an impact on EU policies. About 14% of the 227 projects are marked with “Good innovation potential” but only less than 8% (17 in 227 projects) were highlighted as with Substantial R&D breakthrough character.
Project level results
The TRI-VALUE results at project level essentially derive from the analysis of the production and use of project outputs and the attempt to link these to the steps of the Research Impact Pathway, which will allow conclusions on the impacts and added value of research activities. Moreover, the assessment of results took into account different yet complementary views, as an important component of the analysis was to evaluate whether there are significant differences amongst the transport modes, the project orientation, the budget size and starting date of the projects that would contribute to the results achieved. This is important in the sense that TRI-VALUE does aim to go beyond looking backwards to provide recommendations for future improvement of transport research in Europe.
This analysis revealed that orientation (i.e. whether it was a technology focused or a policy focused project) and mode are the variables that most affect the production of different outputs. Budget impacts particularly on the number of prototypes and pilot scale projects generated while starting date is a variable impacting mostly the number of publications.
Moreover, it is worth underlining that results obtained by TRI-VALUE were based mostly on the results of the online questionnaire to coordinators, referring whenever they differ significantly from those obtained from desk reviews or from Project Officers (i.e. RESPIR). The online questionnaire comprises the answers from 245 projects, representing approximately 41% of the total FP7 transport projects, giving a good level of representativeness for all the transport modes. Regardless of this high response rate, it is also worth noting that within TRI-VALUE multiple sources of evidence are used, allowing for a constant verification and assessment of data consistency.
The main results from TRI-VALUE’s ex-post evaluation are presented in six sections, with the first focusing on the production of outputs by research and innovation projects, the second analysing the dissemination of those outputs and the third looking at the exploitation of those outputs. The following three sections address the development of skills and networks, the contribution to policies and the EU added value of transport research and innovation. Finally there is a section that looks into the achievement of the programme’s objectives.
The Production of Outputs
The first component of the ex-post evaluation of TRI in FP7 looks at the concrete outputs generated by research projects as a whole. Figure 5 represents the share of projects generating a pre-defined set of concrete outputs, according to the online questionnaire.
As can be seen, a very high share of projects (>60%) delivered peer review publications – underlining the knowledge creation pillar of TRI – and of testing activities (validations and verifications) – which may be linked with activities related to development of new products or services. Development of software, tools, models and applications (not marketed) is also very common (i.e. occurs in more than half of the projects). However, only a small proportion of projects (<20%) delivered new services, new products, new norms and standards or patents, which may reflect that fact that TRI in FP7 focused much more on basic research and early stage development rather than closer to market applications.
Importantly, the type of output generated highlights significant differences, as expected, when comparing policy and technology oriented projects and the different modes. The outputs in terms of patents, prototypes, testing activities, models and tools, new technologies is considerably higher in technology oriented projects, as might expected; however, projects technology oriented also produced significantly more peer reviewed publications and PhD theses. Looking per mode it is noticeable that those producing more scientific publications in policy oriented projects belong to the aviation and water sectors, while in technology oriented projects it is rail and intermodal that do, with urban being the mode that maintains a high level of scientific publications both in policy and technological projects.
On the other hand, the number of policy oriented projects reporting the production of Assessments (i.e. impact assessments, consultancy studies, etc.), Handbooks & Guidelines and Policy Recommendations is greater than for technological ones. Per mode, it is interesting that production of assessments is particularly noticeable in water projects, while policy recommendations arose mainly from intermodal and road projects. However, it is in urban projects that this type of output is mostly frequently produced.
An interesting aspect is the fact, that while in general technological oriented projects were more market oriented, the proportion of projects in policy oriented projects reporting the production of new products (i.e. new materials, Physical Parts, Machinery, Facility Infrastructure) and Services (including ICT and non-ICT services) that have been introduced to the market is higher than the technology oriented ones, this being observed for both urban and rail projects.
When comparing for the same type of outputs, those that have produced more than 4 outputs (compared with those that have produced at least 1 output) consistently it is observed that, independently of mode, technological oriented projects (and typically those with longer duration and higher budgets) show the highest results.
Similarly important to the production of outputs is their use and adoption by the relevant communities. Respondents to the survey were asked to state which of the users they had named as important or involved in their research had actually made use of the research results, for instance referring to them in policy documents or by implementing them. Globally for the FP7 projects surveyed, the research community is the user that makes most use of the outputs produced (for about 40% of the projects) and implemented results for about 30%. For the other users this figure is about 20% of projects for reference and use in policies, a proportion that decreases when referring to the implementation of results.
However, as with the outputs produced, its actual use by the different users varies significantly by mode and project orientation. Coordinators were requested to refer to the uses of which they were aware; the main message is that for the majority of projects, coordinators are not totally aware of project results being used, particularly after the end of the project.
The use by EU institutions of policy projects is mostly noticed for rail and water projects, namely its use in policy papers and implementation (i.e. via a Directive or Regulation). For technology projects, the use of results is noticed mostly as “reference in documents”, namely for rail and urban projects.
Local and National Authorities use in particular the results from policy projects, this being visible most in urban projects, at least where coordinators are aware. This can be explained in part by the presence of more local authorities taking a role as participants in projects (i.e. CIVITAS). The use in policies and strategies is also noticeable in air projects, road and water. Coordinators for rail policy projects tend to be unaware of use of their results by local and national authorities, although they referred to their use in some technology projects.
As could be expected, SME’s made more use of technology oriented project results rather than policy, although the case of urban policy projects reveals a degree of use significantly above the other modes. This could be partly explained by the presence of several SMEs acting as consultants at local /urban level using the results of research projects frequently in other studies and projects. In technology oriented projects, implementation of research outputs by SMEs is particularly noticeable for the rail mode, followed by water and road, being slightly lower in air projects, where despite the presence of several SMEs as providers, the large companies still have a stronger role.
Regarding the use of results by large companies there is a trend towards higher use of technology oriented projects, perhaps with the exception of rail, where policy projects are more often used, referred to and implemented.
As noted above the research community is the main user of TRI project outputs. Here the difference between use of policy and technology project outputs is hardly noticeable, underlining the cross-cutting work of the research community.
Dissemination of project outputs
Dissemination is critical to increase the impact of project outputs. There are various ways in which project results can be disseminated: through publications, presentations at workshops or conferences, newsletters, website announcements or project reports. It is impossible to determine the best dissemination mechanism, as it depends on the “message” and the “audience”, accordingly, the TRI-VALUE online questionnaire addressed this issue by requesting respondents to rate the relevance and actual use of different dissemination means for the impact of their projects.
Coordinators were asked to classify the relevance of each dissemination mechanism used in a scale from 1 to 5, where 1 means “very low relevance” and 5 “very high relevance”. The results for the dissemination means classified as very relevant /very high relevance (4 and 5) are presented in Figure 6 according to their orientation and mode.
Globally, the “standard” instruments, i.e. most commonly used in transport research projects, such as websites and events, either other events or events organised by the projects, remain as the ones most reported by a high number of projects. Promotional material and newsletters are very relevant in policy projects and scientific publications for technological projects. Presence in social media is reported in nearly 20% of the projects, independent of their orientation, so it can be identified as a growing trend in transport research dissemination.
Results per mode reinforce the importance of website and events, especially in rail, urban and water policy projects and urban technology projects. Scientific publications are also identified as a very relevant dissemination mechanism for technology projects, especially at the urban level. It is also worth noting that newsletters, which used to be seen as the standard mechanism for TRI European projects are now ranked at values close to those of the brand new “presence in social media”, especially in technology projects. Another interesting element to notice is the pattern that air and road transports rely relatively less on Press Releases for dissemination, while urban and water seem to be making good exploitation of this mechanism both in Policy and Technology projects.
Based on the overall results one can conclude that dissemination is recognized as an important feature for European transport research projects, although different projects tend to use different dissemination mechanisms.
An important aspect to reflect upon is on whether the dissemination strategies put in place by transport research projects are succeeding in engaging the broadest possible public beyond the research community in the debate on scientific issues and research results. By looking at the results on questions on the use of project outputs by the different communities we can conclude that, since the research community is identified as the main user of transport research results and many project coordinators are unable to identify use being made of their outputs outside the research community, there is a lack of motivation for coordinators either to disseminate beyond the scientific community or to identify the use of their research results.
Still, several initiatives in the field of scientific communication and education, including the involvement, where appropriate, of civil society organisations or networks of such organisations, were launched over the program. Efforts have also been made towards targeting the future generation of researchers and professionals in the transportation area. Additionally activities for fostering coordination of research efforts (i.e. CAPPADOCIA, CORE-JETFUEL, REACT); capitalisation of results (i.e. SOLUTIONS, CAPITAL, TIDE), knowledge transfer (i.e. TURBLOG-WW, SEAHORSE, CO3) have been launched in practically all the transport sub-themes. Furthermore, specific actions to setup (i.e. ETNA) and give continuity (i.e. ETNA PLUS) to a network of National Contact Points to support and foster the active participation of stakeholders in European research calls and projects were promoted with large success in reaching a wide universe of stakeholders and in supporting activities.
Exploitation of project outputs
Dissemination and exploitation frequently come together in projects, notwithstanding their different objectives. Exploitation of outputs is market oriented, i.e. it looks to exploitation as a means to progress from the research results towards its deployment and uptake. Previous projects dedicated substantial attention to this issue and on-going projects are focusing on developing support to different consortia exactly on the exploitation aspects.
Exploitation of project outputs was assessed through a combined analysis of the online questionnaire and the interviews with coordinators. It is worth noticing that responses were generally slightly more optimistic in the answers received from the former, than from the latter.
Coordinators of surveyed projects were requested to indicate whether in the context of the project or after its conclusion, projects had developed i) a business plan; ii) direct contacts with stakeholders (i.e. users) or potential clients; iii) established agreements with private investors, and if there was any technology transfer associated with their projects. Figure 7 presents the results obtained from the online questionnaire.
Approximately half of project coordinators surveyed have engaged with developing a business plan and analysed markets. Nearly 90% of them are in direct contact with stakeholders and three quarters made contact with potential clients during the project period. Some of them (~11%) have even contacted with private investors such as venture capitalists. About 4% have succeeded in signing up agreements with private investors. About 40% of the respondents undertook technology transfer activities during the project period, increasing to almost 50% after the project conclusion.
Surprisingly, the views about economic exploitation of project results are more optimistic "during the project" than "after the project". Actually, within a conventional project cycle perspective, exploitation of results should be more interesting and promising once the project has concluded. It could be considered here whether these answers reflect some lack of commitment and interest about exploitation of project results by some coordinators, once the project is completed; this would indicate a significant barrier to implementation and a crucial failure within the innovation process.
Direct contact with stakeholders is highlighted as the more frequent action and result of a project from an exploitation perspective (93% of answers in the questionnaires and 87% in the interviews). Regrettably, the number of positive answers decreases significantly (to 76% and 59% respectively) once the project is completed.
Contact with potential clients is also pointed out by many coordinators as one of the exploitation-related activities undertaken. It is worth noticing that there are only small differences between the percentage of positive answers referring to the activity during the project and after the project. In the on-line questionnaire, these percentages are, respectively, 75% and 68%, and during the interviews, 62% and 40%.
The rest of the activities suggested by the questionnaire and the interview received much lower positive answers. The development of a business plan and/or market analysis nonetheless received a significant number of answers in the questionnaire (54% during the project and 34% after the project), but much lower ones in the interviews (31% and 12%). Technology transfers are mentioned by 30% (during the project) and 19% (after the project) of questionnaire responders, and by slightly lower percentages in the interviews (25% and 8%, respectively).
Contacts with private investors (VC, Business Angels...) and agreements with private investors, both during and after the project, are pointed out only by a small percentage of the responders both, in the on-line questionnaire and during the interviews.
Finally, it is worth noticing that no concrete figures were provided on the estimated market value of projects’ results. Actually, only two coordinators provided some detail (€300,000 in the case of one small CSA focusing on educational issues and an "extreme large value" (with no further detail) in the case of one rail project. These results provide some evidence about the need to dedicate more attention to the exploitation potential of project results, and the contact with key stakeholders for facilitating this.
The policy orientation of projects looks not to be exerting a major influence on the exploitation of results with the exception of contacts and agreements with private investors, namely in water policy projects. In technology oriented projects contacts with private investors are rarely indicated by coordinators in the online questionnaire. This could result from the presence of industry as part of the consortia.
The development of skills and establishment of networks
Projects often contribute to the development of effective cooperation and networks. The access to complementary expertise and “network” effect from FP7 projects is one of the most intangible assets generated by transport research. While the benefits of such assets are hard to quantify, having several informal or (formal) networks with members from different communities (research, industry, consultancies, NGOs) and countries is valuable. The involvement of junior researchers in projects is also frequently reported as an important contribution of research projects towards academic excellence.
In TRI-VALUE, coordinators were asked to indicate whether projects:
• projects led to the creation of a formal network under the scope of the project
• there was cooperation with other partners of the consortium after the project completion
• there was cooperation with other external organisations as result of the project´s activities
More than a third of projects have been successful in creating a formal network within the scope of the project, and an overwhelming majority of them (>86%) continued to cooperate even after completing the project for which they originally formed a consortium. Nearly 75% of the respondents also developed contacts with external organisations as a result of the project.
The analysis performed in TRI-VALUE showed that cooperation after project conclusion is mode independent and a good example of the networking effect of the collaborative research in the EU FP’s. In more than 80% of projects (exceptions are water policy and road technology projects, but still outline a high share of responses) follow-up collaboration occurred. As a result, collaborative follow-up with members of the consortium or with external organisations is also not really influenced by mode or project orientation.
However, transport mode and policy orientation seem to exert some influence in relation to establishment of formal networks, compared with technological oriented projects. While there are several examples of networks established from FP7 projects, this seems to be a particularly relevant result in water policy projects, with nearly 75% of projects in that category referring to this type of networking. In the opposite direction, for air projects, formal networks look to be a relevant result in technological projects, but none of the air policy project coordinators reported such a type of collaborative effect. For remaining modes, no significant differences relative to their orientation were noticed.
Contribution to Policies and Environmental, Social and Economic Impacts
The potential contribution to policies emerging from FP7 transport research activities was addressed by TRI-VALUE in the online questionnaire the interview and the desk review. In the online questionnaire, coordinators were asked whether their project contributed, directly or indirectly, to transport policy objectives. The comparison between the information provided by coordinators is generally coherent with the evaluation from TRI-VALUE desk reviews. Results are outlined in the Figure 8.
In general there is a quite positive evaluation among the objectives identified, in particular for GHG reduction and safety, followed by pollutants reduction and energy efficiency. At the other extreme, there was little attention to recycling measures.
On the energy side there appears to be much more focus on energy efficiency than on finding new fuels or energy sources, which may reflect the fact that so far regulations have focused on targeting directly emissions, which are closely associated with energy efficiency
Across all modes there is a large share of projects contributing directly or indirectly to those objectives.
It is clear that the modes with a higher contribution are Road and Urban, which may also reflect the high share of road-based mobility within transport CO2 emissions and/or the high regulatory pressure that was observed over the last few years.
At the other extreme, we have rail, perhaps because rail transport has a marginal contribution to CO2 emissions and is already among the most efficient modes of transport.
Policy orientation of projects exerts some influence in terms of a more direct or indirect contribution to the objectives, with technology oriented projects showing a more direct contribution, namely for emissions, noise and safety.
From the coordinators’ perspective, urban projects emerge as the ones with a direct contribution to the reduction of pollutant emissions, this being consistent for both policy and technology oriented projects. This fact can’t be disassociated from the fact that urban policy projects are aligned with the clean urban package objectives and EU initiatives such as the Covenant of Mayors. Technologically oriented projects on average have a more direct contribution to the objective than policy ones, as might be expected.
In general, projects show mainly an indirect contribution to reducing congestion, both for technological and policy oriented projects. The exception is urban policy projects with nearly 60% of the answers showing a direct contribution to this objective.
Security looks to be an objective not so strongly considered in transport projects or where projects’ contributions (either direct or indirect) are less evident; however the presence of a dedicated program on “Security” might partly explain this.
In what refers to Environmental, Social and Economic (ESE) Impacts, it is worth starting by underlining that while most FP7 transport projects might have wider societal impacts in some way or another, in the overwhelming number of cases those impacts will only arise in the long term. Accordingly, the results from this ex-post assessment shall be read with the limitations associated with the timing of the analysis in mind.
Indeed, topic descriptions in the FP7 working programmes routinely made reference to ESE objectives, and most of the FP transport projects may claim to have wider societal impacts in one way or another; however, in an overwhelming number of cases those impacts will only materialise in the long term. There is an inherent difficulty for coordinators (and also for evaluators) to substantiate how one particular research project can result in concrete ESE impacts.
In fact, it could be claimed that these impacts are primarily considered in the stages prior to actual research: firstly, by EC officials during the preparation of the topics in the working programme and secondly, by researchers during the preparation of their proposals. At both stages, there is some elaboration on how the research activity can contribute to policy objectives and produce particular ESE impacts.
However, no evidence was found about the active monitoring of these impacts during the deployment of research activities or at the project completion stage. This may explain the general lack of evidence found during desk reviews and interviews by evaluators regarding concrete ESE impacts from projects.
Macroeconomic impacts include effects of transport projects on macroeconomic variables (i.e. employment, consumer price, regional effects, international trade, and public authority budget). Therefore the indicators used for tracing these impacts were:
• Job creation after project development
• Consumer prices
• Differentiated impact on specific regions
• International trade
• Impact on public authorities’ budget
• Overall macroeconomic impact
Most assessment of macroeconomic impacts was based on the desk reviews, with the exception of job creation, which was assessed in three different ways:
• Through the on-line questionnaire completed by the coordinators (using a qualitative scale: no, yes- indirectly, yes-directly).
• During the telephone interview with project coordinators (using a quantitative 0-10 scale).
• During the desk review (using the same quantitative 0-10 scale).
Around 30% of projects contributed to job creation. The comparison of three ways to detect the job creation contributions shows a certain consistency. Nevertheless, few coordinators are aware that their projects create new jobs or improve the quality of jobs. This was an issue generally present in the approach of the European Technological Platforms with some differences among transport modes: ERTRAC and Waterborne platforms highlighted the problem of job quality in their sectors and they are looking for improving the conditions of their jobs and their attractiveness. Air emerges as the transport sector where more progress has been made in terms of improving job quality (e.g. the HUMAN project which analysed the design of aircraft cockpits).
In general, less than 20% of transport projects contribute to stabilize consumer prices and 16% seems to contribute to balance of differentiated regions. 30% of transport projects had impacts on international trade and 21% contribute to lower public authorities’ expenditures.
Impacts on key macroeconomic priorities at EU level (i.e. job creation, stabilizing consumer prices, stabilising public authorities’ budget and general macroeconomic growth) seem to be limited or, at least, not easily detectable in the transport projects.
Between 20% and 30% of transport projects were said to contribute to key macroeconomic priorities. However, coordinators think their projects have major impacts at macroeconomic general level.
Other economic-related priorities are addressed by a higher percentage of projects: 75% of relevant transport projects increase EU competitiveness and more than 50% boost the EU economy. According to the desk review, only 16% of transport research projects would contribute to improve the quantity and quality of jobs, in spite of the general contribution to "job creation" at the macro level claimed by 30% of projects.
In terms of environmental impact, the picture framed by TRI-VALUE analysis shows a modest direct contribution, although figures increase significantly when indirect contributions to environmental improvement are included. For example, over 80% of projects claimed to contribute directly or indirectly to reducing greenhouse gas emissions, 75% have contributed to improving the local environment and almost 50% to the development and use of sustainable/renewable sources of energy. To strengthen the environmental profile, better identification of the connecting paths between transport research and the environment, and more directly addressing environmental dimensions during the definition and selection of research topics could be helpful.
Direct social impacts associated with the FP7 transport projects were rather limited, outside the urban modes. Linking this to consideration of key social priorities at the EU level, few transport projects appear to address basic social services, poverty and social exclusion, although causal links are already known. Ageing is another domain European Transport Research could pay some additional attention to, as is also the case for security issues relating to user perceptions and attitudes; both are expected to further gain relevance in an ageing, multicultural society with a pervasive risk of segregation.
There appears to be a need to better identify the connecting paths between transport research and the social and environmental impacts, and to address these dimensions during the definition and selection of research topics. The implementation of a permanent tool for projects follow up (i.e. ”TRI research watch”) could support the ex-post evaluation.
However, the ESE evaluation undertaken by TRIVALUE suggests that the much needed socio-environmental transition mentioned in EU policy papers and ETPs' Strategic Research Agendas is still to be achieved.
An additional analysis would be needed in the future in order to gain deeper understanding of the barriers and possible pathways for increasing the short and medium-term ESE impacts of research. We suggest adopting in the future a more structured, interdisciplinary approach for the definition and assessment of transport research priorities and working programmes, including the various levels involved in the innovation cycle (niche-innovation, the prevailing socio-technical regime and the socio-technical landscape in a more balanced way. Niche-innovation represents the locus for radical innovation; socio-technical regime includes established industry and current technology and habit of users’ practices; socio-technical landscape encompasses landscape development, (i.e. spatial structures, political ideologies, societal values, beliefs, and macro-economic trends). This approach, should facilitate the identification of desired ESE impacts in the short and medium-term, and the involvement of researchers in the pursuit of those impacts during the project.
Socio-environmental transition is characterized not only by technological changes that in the transport sector, usually categorised as “mature” (in the sense that no radical changes of technological paradigms are expected, compared to sectors like ITCs or biotechnologies) are quite marginal, but specially by changes in users’ practices, definition of new regulations (i.e. land use patterns management), introduction of new mobility services dealing with low cost systems of private and public transport (car sharing, carpooling, urban bike systems…), etc. Differently from environmental transition of transport systems, more oriented to technological changes (i.e. between fuel and electric vehicles), socio-environmental transition is oriented to analyse changes in user practice and lack of regulation where EU TRI should play a key role in transition. Socio-environmental transition is holistically based, and points to the social inclusiveness of transport systems in the light of EU environmental objectives.
Added value of EU scale research funding
EU added value is variously acknowledged to be a concept whose concrete application is unclear (Zuleeg and Mlino, 2011) and relatively undefined (Culture Action Network, 2011). Nevertheless, EU added value in relation to research and innovation is of great interest and importance in seeking to understand the role and impact of the EU in this area.
The TRI-VALUE results highlight that there is high EU added value from the activities within FP7 and that most coordinators attribute this value to aspects such as the “network affect” access to complementary expertise, or to overcoming barriers for uptake of results.
Such an asset is hard to value, but there are clear advantages from having several informal networks with members from different communities (research, industry, consultancies, NGOs) and from different countries.
The assessment of the added value results primarily from the interviews with coordinators for the sample projects, being questioned about how they evaluate the added value of EU research funding in terms of:
• Access to complementary expertise
• Understanding the needs and characteristics of a wide market
• Overcoming specific barriers to the market uptake of research results
• Overcoming lack of funding at national level
• Capitalising on previous EU projects experience and networks
• Availability of alternative sources of funding
In general the results are consistent within each single mode.
“Different cultures, with different levels of expertise... it was not possible to conduct this type of project accounting just with one country… the global perspective would be lost!”
In the aviation sector, all the factors were assessed to be important among the projects surveyed. In particular, access to international complementary expertise and overcoming lack of funding at the national level were important in 80% of the projects surveyed. This shows the critical importance of EU funding in air research and technology development.
“Without European funding this project would not be possible to be carried out.”
In intermodal projects, the assessment of added value indicates a focus on understanding the needs of a wider market, though also with strong views on the access to complementary expertise followed by capitalising on previous EU experience and overcoming the lack of national funding.
“The international consortium work enables to verify and think in broaden terms.”
The main added value, stated by all respondents on the rail projects is the access to complementary expertise. Two thirds of the respondents give high scores to the added value of "understanding a wider market", "overcoming lack of funding" and "capitalising on previous EU project experience".
“Took advantage of the knowledge that has been created in Europe and transfer of this globally”
“Results were given to industry and plans made for more industry participation in the next project and plans made for developments subsequent to the next project including exploitation of follow-on project results”
For road projects, access to complementary expertise is also the key added value of EU research. Capitalising on previous EU project experience obtains the second highest score, while two thirds of respondents also note the added value of "understanding a wider market" and "overcoming lack of funding".
“Helped to open new horizons, invitations to other commercial events and be further recognized in the area.”
For urban projects added value of EU scale research is almost equally spread along all categories, with the exception of availability of alternative funding. Access to complementary expertise and capitalising on previous research are however the two main added value areas pointed out by all the coordinators.
In conclusion, evidence reviewed in TRI-VALUE demonstrates considerable added value from European research, as it contributes to promoting excellent science in transport at the European level.
Firstly, it shows that the international consortium work enables researchers to verify and think in conceptually broadened terms. Secondly, in some areas the project topic was highly international (maritime, air). Significant numbers of projects could not have been undertaken without European funding and in some cases, adequate funding was simply not available at national level.
Whilst added value comes from the collaboration across borders and experiences and skills which contribute to a more comprehensive piece of research, national research funding rules often do not permit funding of international projects. For example, it is clear that the added value in terms of helping the European transport industry to lead at the international level is particularly noticeable in major maritime and air projects with partners such as Korea, China and USA. Furthermore, given the complex and global nature of air-transport, often it is mentioned that only European funding could have supported this kind of research.
Achievement of Program Objectives and Impacts
A critical element in any ex-post assessment of a research programme is to determine whether the objectives stated in the work programme and the expected impacts referring to each specific funded project were achieved. These execution indicators were explored in TRI-VALUE through an independent analysis in the sampled projects (i.e. an assessment by the TRI-VALUE project reviewer), that was later discussed with the coordinator in the telephone interview, which allows an overview of performance in this area. The results from this analysis are outlined in Figure 9.
These results show that on average transport FP7 project objectives reflect the work programme and call objectives, with an average assessment of 4.7 (maximum of 6) and more than 90% of the projects ranking more than 4 (i.e. having a positive grade). In terms of the level of achievement of project objectives and expected impacts the results are somewhat lower, with average grades being 4.5 and 4.2 and with fewer projects above 4, namely 83% for the achievement of objectives and 80% of the expected impacts.
Looking at results per mode of transport and project orientation TRI-VALUE concluded that, with the exception of road transport, there are no major differences between the assessments for policy or technology oriented projects. In short this means that we cannot observe a pattern of project orientation influencing the assessment of the achievement of objectives or expected impacts. As for the influence of modes of transport the average assessment grades vary from 4 to 5 (in a scale of 0 to 6) with the exception of rail projects which seem to score lower grades.
The assessment of the achievement of objectives was checked with coordinators in the telephone interview. Major disagreements in the assessment were not found.
This data suggests a generally very good achievement of programme objectives and a good achievement of expected impacts. These reassuring results do not mean that problems do not exist –roughly 20% of projects still show a negative evaluation on achievement of expected impacts.
In order to improve this performance the European Commission could study the implementation of a more thorough ex-post assessment of projects which could range from a grading of performance by the project officer that could be used to profile participants (and applied in future projects evaluation) or the implementation of quality audits for assessing achievement of objectives and expected impacts inspired by those already performed for project financials.
Another important indicator related to the execution of the FP7 transport research programme regards the involvement of SMEs in research activities. SMEs represent a major share of companies and account for most jobs in Europe. Additionally, SMEs are a source of innovation and they conduct a large amount of R&D, but due to the characteristics of research in the Transport sector, the access of SMEs to innovation is not always easy. This issue was deeply debated within the Market-up project, which concluded that “there seems to be an untapped reservoir of innovative transport SMEs to participate in research activities, and there are numerous studies, such as the Eurogia+ programme, noting the importance of the right combination of large enterprises, SMEs and research institutes, as each type of actor brings valuable features to the projects.” (Market-up Project, 2012)
Accordingly, several rules and operational procedures designed to ensure the programme is relevant to SMEs with the necessary domain knowledge and in-house research capacity were included in FP7. The two most important SME features followed were:
• a target of 15% FP7 funding to go to SMEs in the Cooperation Programme;
• a 75% funding rate for SME participants, compared to 50% for large companies.
While all topics are open and recommend the participation of SMEs, actions oriented towards supporting the participation of SME and fostering their role were launched specifically, along all the three sub-themes of transport program. The 15% objective for SME participation has been reached across all the research activities: SME participation overall is 25%. Moreover, as seen above, SMEs take 21% of the total budget and have high shares in terms of coordination responsibilities.
Other studies have looked more specifically at transport-related SME engagement, notably the FP7 Market-up on Transport Research Market Uptake. Within TRI-VALUE itself, a range of analyses were undertaken, focusing in particular on whether SMEs were identified as end users and users of outputs. This analysis is reported in Deliverable 2.1. In summary, the data suggests a fair degree of SME engagement with, for example, 42% of project coordinators in the on-line survey identifying SMEs as potential users of project results while 33% of them have implemented their project results. This general picture is repeated for the other modes examined, although with slightly lower levels in the road sector.
These results can lead to the question of whether the particular interests and needs of SME were duly considered in the elaboration of the work programs. Such concern has been taken into account with a set of call topics addressing specifically SME participation (see below in the table the targeted SME actions). However, the data gave only limited insight into the scale and nature of SME involvement and little relating to topics such as the effectiveness of measures to encourage SME engagement in the short term. Success of projects targeting the support for SME participation will probably only be visible within subsequent programmes.
Another relevant indicator refers to stakeholders’ engagement beyond the research community. The fact the Commission pushes for appropriate dissemination of project activities has impacts in the call for proposals, in the evaluation and negotiation process and in its project monitoring and has certainly contributed to the significant numbers obtained and the potential success story of dissemination activities. A major question is whether those dissemination actions were also able to reach and engage stakeholders beyond the research community in the debate on scientific issues and of research results. Several initiatives in the field of scientific communication and education, including the involvement, where appropriate, of civil society organisations or networks of such organisations, were launched over the programme.
Efforts have been also made towards targeting future generation of researchers and professionals in transportation. Some of those examples include the projects EDUCAIR and, SKILLRAIL in the fields of assessing educational needs and shaping a new generation of workers. Other projects target specifically actions to attract future careers (i.e. FLY HIGHER) and to raise awareness of potential job opportunities (i.e. TECH-CLINIC SST, PROMARC).
Additionally several initiatives towards the coordination of research efforts (i.e. CAPPADOCIA, CORE-JETFUEL, REACT); capitalisation of results (i.e. SOLUTIONS, CAPITAL, TIDE), knowledge transfer (i.e. TURBLOG-WW, SEAHORSE, CO3) have been launched in practically all the transport sub-themes.
TRI-VALUE Conclusions and Recommendations
Considering the results outlined in the sections above, which were discussed with stakeholders and Commission staff at several occasions throughout the project, TRI-VALUE raised a number of conclusions and recommendations aiming to derive messages from this assessment to build constrictive links to improve Horizon 2020.
In total 13 Conclusions and 8 Recommendations were prepared, which are briefly presented below.
Conclusion 1: The Transport Programme shows a large contribution to the achievement of the FP7 objectives and of the “Cooperation” specific program
TRI-VALUE results estimate that 54% of projects developed in FP7 made a contribution to strengthen the competitiveness of the European industry and more than half of all transport research projects developed in FP7 have contributed towards the achievement of increased efficiency of the whole transport system.
In general, there is good alignment between the work conducted under FP7 and the transport policy objectives. In particular, GHG reduction and safety were key focus areas for transport research, followed by reduction of pollutants and energy efficiency. Across all modes there is a large share of projects (i.e. 75%) contributing directly or indirectly to those objectives.
All in all, more than a quarter of policy related projects and close to one fifth of the technology related projects anticipates their outputs to be used by the European Institutions. This, together with the finding that 60% of all projects deliver policy recommendations, materializes a substantial contribution of FP7 to the European transport policy making process.
Conclusion 2: The FP7 Transport programme reveals a strong European Added Value
The analysis of EU added value has demonstrated that many activities would not have been performed without funding from the European Commission. This is reported to be particularly true for aviation and shipping, which are by nature international, and which would be disadvantaged if they had to rely exclusively on national funding for R&D activities.
A second important piece of evidence is the creation of networks of research across Europe. The access to complementary expertise and “network” effect from FP7 projects is one of the most intangible but important assets generated by the transport research. The TRI-VALUE analysis showed that more than a third of projects have been successful in creating a formal network within the scope of the project, and an overwhelming majority of participants (>86%) continued to cooperate even after completing the project for which they originally formed a consortium. Nearly 75% of the respondents also developed contacts with external organisations as a result of a project. This reflects a great support and considerable success by FP7 in building skills and knowledge networks for transport research across Europe.
A third aspect of relevance on the contribution to a knowledge-based society refers to the engagement of the general public in the debate of scientific issues and research results. Although the research community remains a key player in transport research, the analysis shows that several projects have benefited and have derived part of their success through conveying
Conclusion 3: FP7 Transport research shows a good level of achievement of project objectives and expected impacts set out in the annual work programmes
The TRI-VALUE results showed that, on average, transport FP7 project objectives reflect the objectives of the annual work programme and respective calls. On a project basis, there is a good level of achievement of objectives and expected results, with 83% of the projects scoring more than 4 on the achievement of objectives and 80% for the expected results (On a scale of 1 to 6,).
Conclusion 4: The framework for cross-cutting activities could be further developed
The approach adopted for FP7 was an outstanding improvement compared to the previous FPs, including aspects of programme design in terms of main objectives and sub-themes, specifically covering cross cutting activities.
In addition to the TPT programme, which generally addresses multidisciplinary aspects but only accounted for 3% of the budged, cross-cutting topics were included within the modal components of the annual work programmes. This resulted in a somewhat fragmented landscape for cross-cutting topics, partially due to the underfunding of this dimension in the initial FP7 design stage. The approach to cross-cutting activities (including in particular socio-economic, institutional and decision-making issues within the transport sector, which are of cross-cutting nature), however, lacked an appropriate framework, comparable for example to the Strategic Research Agenda of ETPs for modal research. This made it difficult to understand the logic behind topic selection and to properly develop adequate research and dissemination strategies by researchers.
Moreover, the increased attention to intermodality (e.g. one of the major ambitions of the Transport Policy White Paper is optimising the performance of multimodal logistic chains), which is cross-cutting in nature, shall drive a reflection on whether the budget distribution in FP7 , with only 3% of the budget devoted exclusively to cross-cutting issues, may fall too short. Increasing funding in this category within Horizon 2020 is thus recommended.
Conclusion 5: Procedures for priority setting and Work Programme design are well established contributing to the realisation of ETP Strategic Agendas
Annual work programmes are adopted by the European Commission, with the assistance of the Programme Committee. They set out information on the priorities, objectives and policy relevance of the research topics, which would be implemented through calls for proposals in that specific year. In preparing the work programme, the Commission relies on advice from a wide range of stakeholders, including from the European Technology Platforms (ETPs) and the Transport Advisory Group (TAG). Program Committee, with the representatives of Member States, is then responsible for giving final approval of the work programmes.
European Technology Platforms main contribution is to provide the strategic orientations to the work programme design. Strategic Research Agendas present the approach adopted by each Technology Platform to establish research objectives and development priorities, timeframes and action plans (Roadmaps) that tackle the challenges of the respective industrial sectors. The regularly updated versions of the Strategic Research Agendas are important input to the design of the work programmes.
However, based on the assessment of technology-oriented projects (which are the ones usually aligned with the ETP's SRAs), it can be stated that, in spite of a satisfactory result in terms of the achievement of objectives, the answers by coordinators on the exploitation of results reflect a lack of awareness on the use of project outcomes. For example, 50% of coordinators “don’t know” of SMEs that directly implemented project results and/or recommendations and 49% of are also unware of such use by large companies. Moreover, only two coordinators provided some detail on the estimated market value of projects’ results, and one of them was unable to quantify. This could hamper FP7 attempts to move closer to innovation, development and market implementation with the support of ETPs.
Conclusion 6: Transport research provided substantial contribution to the formulation, implementation and assessment of Community policies
With more than a quarter of policy related projects and close to one fifth of the technology related projects producing outputs to be used by the European Institutions, with the indication that 60% of all projects delivered policy recommendations and with project coordinators estimating that 15 to 30% of projects already produced results that were “used to date” by the EU institutions, it is concluded that FP7 activities have a substantial contribution to the European transport policy making process.
The TRI-VALUE evaluation is clearly indicating that in terms of outputs and dissemination, projects are making an effort to be aligned with and contribute to the policy processes. An analysis of whether this effort is reflected in concrete Community policy outputs is very difficult to perform. The research and innovation process has its own long term time-frames which impacts the effectiveness of an ex-post assessment just after the end of a project and programme. This adds to the long timeframe of the policy formulation processes, which is especially true at European level.
Conclusion 7: Transport research has provided initiatives to engage the public, beyond the research community, in the debate on scientific issues and research results as well as in the field of scientific communication and education
The picture in terms of project participations shows a good balance between industry (being large companies or SMEs) and research (being research bodies or educational institutions). Still, this pattern reflects a certain need to improve the participation of civil society organisations, which may play a very important role both in involving the public beyond the scientific community, but also in improving the contribution to policy formulation. Analysis shows that projects have been successful in conveying and disseminating messages beyond the scientific community, especially towards business organisations.
More than 20 projects were specifically designed to target communication beyond the research community. Specific actions to setup and give continuity to a network of National Contact Points to support and foster the active participation of stakeholders in European research calls and projects have been promoted, with large success in reaching a wide universe of stakeholders and in supporting activities.
Conclusion 8: Transport research shows an adequate adoption of measures for dissemination, knowledge transfer and broader engagement
Dissemination of the results of the work undertaken in FP7 has been strong, through conventional conference papers and peer-reviewed journal publications, but also through electronic means. Websites are seen as particularly important, but there are also signs of increasing use of social media for dissemination purposes.
Several initiatives in the field of scientific communication and education, including the involvement, where appropriate, of civil society organisations or networks of such organisations, were launched during the program. A good number of projects is targeting the future generation of researchers and professionals in the transport sector focusing on coordination of research and knowledge transfer, in practically all the transport sub-themes.
Conclusion 9: SME involvement in Transport research largely exceeds the threshold established for FP7
SMEs in Transport research represent 24.5% of project participants (reaching 28% in CSA projects) accounting for 21% of the EC funding in FP7. These data suggests a good degree of SME engagement, which adds to the fact that 42% of project coordinators in the on-line survey identified SMEs as potential users of project results while 33% of them have implemented project results.
The particular interests and needs of SME were considered in the elaboration of the transport work programs, with a set of call topics addressing specifically the SME participation and /or supporting their activities in the program. Data obtained provided only limited insight into the scale and nature of SME involvement and little information relating to topics such as the effectiveness of measures to encourage SME engagement in the short term, and success of projects targeting the support for SME participation. These elements will eventually be addressed in the next programs.
Conclusion 10: Collaborative research in transport allowed the creation of networks of excellence on transport research
As noted above, more than a third of the projects have been successful in creating a formal network within their scope and an overwhelming majority (>86%) of them continued to cooperate even after completing the project for which they originally formed a consortium. Nearly 75% of the respondents indicate that they have developed contacts with external organisations as a result of the project. These numbers confirm the perception, gathered in stakeholder consultation processes, of the importance of this aspect. Being part of an international consortium offers positive spill-over effects widely recognised but not easily measured.
Conclusion 11: The number of International cooperation actions supported has increased but many challenges remain in this area
A new approach to international collaboration was introduced in FP7 integrating the international dimension into each thematic area of the Cooperation programme. Globally, throughout the program, more than 200 international participants were involved and 118 projects that included international participants were funded. Four joint calls resulted in 12 funded projects.
The evaluation showed that challenges remain in this area. It is often unclear what the added-value of the engagement of international partners is and the extent to which this allows, for example, the establishment of networks at global level similar to those that are being established at European level.
Conclusion 12: Transport research responds to transport societal challenges
The analysis has shown that transport research is responding to transport societal challenges, although in different ways across modes.
Globally Climate Change remains the challenge that attained more attention, but with major variances across modes (meaning that more attention may still be needed in some cases). On the other end, security, for which less than 20% of the projects are reported to have direct impacts, appears to deserve more attention across all modes. However, it is important to note that “Security” is one of the topics covered in a dedicated theme within the FP7 ‘Cooperation’ programme, which may justify lower attention within the ‘transport’ theme.
Improving the links between transport research and societal challenges should be better addressed, however, it is clear that a very good basis exists and substantial work has been undertaken and is now being developed in Horizon 2020.
Conclusion 13: Transport research produced positive impacts on innovative activity
The evaluation showed that projects have been successful in producing positive impacts on innovative activity in Europe. Approximately 20% of projects have led to patents, pointing an intention to exploit research results commercially. Participation in projects also enhances the technological development, notably through development and use of new tools and techniques, the design and testing of models and simulations, the production of prototypes, or demonstrators and pilots.
The evaluation outcomes reinforced the need to establish new mechanisms for looking at how the innovation system is functioning along the Technological Innovation System (TIS), providing a framework to assess how effective transport’s TIS is.
On the basis of the above conclusions, TRI-VALUE proposes the following recommendations:
Recommendation 1 – Maximise the contribution from research projects to transport and other policies and establish guidelines/guidance on how to address policy objectives during the research and innovation cycle
Research and transport policy objectives are typically expressed in general terms. They need to be further clarified, so as to be translated into tangible objectives relevant to the different levels of the research cycle. This can be addressed in Horizon 2020 throughout the work programmes (and particularly within the topics' description), the evaluation of proposals and the implementation of the research project. Any sensible monitoring of objectives would need the development of a simple set of performance indicators for each project, to be subsequently assessed upon project completion. This approach would serve to further integrate key policy objectives within the research activity.
TRI-VALUE observed the high contribution of transport projects in the formulation, implementation and assessment of Community policies. Notwithstanding the relevance of that contribution, there is a need to improve/strengthen feedback channels between transport research and EU policies. An obvious way to do that would be through reinforcement of current internal mechanisms of cooperation within the European Commission's relevant services. Furthermore, it can be considered that, for research activities to be more influential in policy formulation, there is a need to revise the relationship between the two. Rather than expecting that research should "underpin" and facilitate policy implementation, a more ambitious role could be envisioned: exploring and identifying "future challenges", widening the policy options available and better integrating transport within key transitional processes in the European society. As such, besides a source of answers, research could recover its role for identifying the right questions. This could be effectively undertaken by including transport issues in the Foresight processes being developed by the EU, for example in cooperation with the European Forum on Forward Looking Activities (EEFLA)
Recommendation 2 – More emphasis on intermodal issues
In order to better address societal challenges, more attention could be put on intermodal transport.
Although some cross modal issues were addressed in the modal categories, achieving “integrated transport” is a major societal challenge the importance of which does not seem to be properly reflected in the budget allocation for cross-modal and cross-cutting issues. Re-balancing the expenditure to better meet the policy relevance of each aspect is a potential area for improvement in Horizon 2020.
In practice this may imply a redistribution of the budget allocation from a mode-by-mode basis and based on the actual contribution of each mode for European transport and/or for addressing societal challenges. One concrete aspect that should be subjected to debate is whether the mechanisms for share of funding between modes should more directly reflect some of the policy orientation (e.g. share of GHG emissions). A careful analysis is recommended, notably looking at future trends and dynamic aspects, such as industrial competitiveness, knowledge and technology transfer across modes.
Effective support to the integration of transport systems would require increased communication among modal research systems, currently organised through ETPs, mainly with a modal structure. Consistence among modal SRAs should be analysed with view of giving due attention to integration issues. Furthermore, a strengthened focus on social, in addition to purely technological issues within SRAs should be included.
Recommendation 3 – Improve conditions for the uptake of transport research results
Despite the positive results obtained in terms of the production of outputs and exploitation of results, a relatively large number of project coordinators did not appear to know what happens to their project outputs after the end of the contract period. This observation suggests a need to focus more on the uptake of transport research results.
This may require an effort in three directions: (1) better identifying those topics with more potential for exploitation, during the preparation of the work programme by the European Commission; (2) better assessment of the exploitation potential and interests of the research team during the evaluation and selection of proposals, and (3) more precise exploitation plans and actions during the implementation of the research project. Across all these phases, detailed exploitation plans may be needed with concrete targets established in 1-2-5 years (i.e. this could be up to 5 years keeping the same metric as the financial obligations). An improvement in this direction may be observed across FPs, following the high priority that has been attributed to dissemination at all levels. Establishing a feedback loop with reporting mechanism on achievements in 2-3 years after project conclusion should also be considered. In addition dissemination efforts could be re-oriented towards a more selective approach addressing best performance in research and highest exploitation potential. In fact current dissemination approach is equitable but focused on itself and, consequently reported by project coordinators very effective. However, dissemination can be a powerful channel for market uptake if streamlined to achieve that goal. This can be addressed within the changes introduced by H2020
These changes are not straightforward, given the fixed-term nature of research contracts, the fact that individual’s move on to other responsibilities, and that research impacts may take many years to surface. However, this would enable a more realistic assessment of economic indicators. The question of better exploitation and reporting feedback has been consistently referred to along evaluation projects but also in projects dealing with the take-up of research results, which makes it a point where the European Commission should devote some attention on.
Recommendation 4 - Improve measurement of impacts and quantification of benefits
The estimated market value of the products or services developed (economic exploitation of outputs) is not known for the majority of the projects. While this may reflect the fact that some coordinators are not in touch with potential end-users, in the TRI-VALUE analysis nearly no concrete figures were provided, with only two coordinators responding to this question. Regardless of all limitations discussed throughout the document, it is hardly acceptable that projects report on the development of products and services with funding from the European Commission and fail to quantify the market value of such products and services as this is a major element for feasibility of exploitation.
Similarly, improvements are needed in the measurement of impacts of transport research activities. Macroeconomic impacts (i.e. employment, consumer price, regional effects, international trade, and public authority budget), the contribution of each research project to the key environmental priorities at EU level and social impacts of projects were constantly addressed in TRI-VALUE questionnaire and telephone interviews. Nevertheless, current data (in final technical reports) doesn’t allow for a quantification of these impacts and gives little room for validating the coordinators’ assessments. Environmental data is not even part of the final reporting templates. Current data is focused on outcome and, at the limit, results can be assessed but impact assessment requires time duration for maturation of effects.
One mechanism to consider is to use “project audits” inspired by those performed for financial assessment purposes. Such audits could assess both technical compliance aspects (i.e. did the project achieve the objectives and expected impacts, whether deviations are justifiable), verify the data included in final reports and promote a thorough assessment of project longer term impacts and benefits. Moreover, specific performance indicators may be added at the proposal stage to facilitate these ex-post audits. The experience of the Intelligent Energy and Marco Polo programmes where expected impacts are estimated in the proposal phase and progress on achievements is reported annually (i.e. x% reduction in CO2 emissions, x% increase in pax.km or ton.km) may be built upon. As indicated above, the identification of such indicators should ideally be supported by more concrete guidance on the research and transport objectives addressed by the relevant work programmes of Horizon 2020.
In addition ex-post assessment should be also performed at later stages, when impacts can effectively be measured.
Recommendation 5 - Establish a Framework to assess the effectiveness of a Technological Innovation System
The scientific community is invited to improve the framework to assess the innovation systems. The Technological innovation System (TIS) approach has been developed to address the actual limitation on assessing how the innovation system is functioning and how effective a TIS is. R&D policy can usefully address weaknesses in these functions. TIS approach is described in Bergek et al. (2008a, 2008b) and has been used by JRC in the mapping of EU transport innovation and in the FP7 Market-up project. They introduce the idea of seven “functions” of an innovation system:
1. Knowledge creation: how the technology can be applied to develop marketable products, performance of the technology, new production processes
2. Guidance of search: what external factors promote or inhibit innovation and which aspects – e.g. performance, safety, environmental impact are externally imposed on innovators?
3. Entrepreneurial experimentation: innovation activities in firms to develop the new technologies.
4. Resource (financial, human) mobilisation: application of investment capital and humand resources in innovation and product development.
5. Legitimation: acceptance in society, adoption of suitable legislation and standards.
6. Market formation (strength of demand): development of new markets or presence of the new technology products in existing markets.
7. Development of externalities or knowledge diffusion through networks: the strength of sharing the knowledge that has been created in innovation networks.
The impact measurement mechanism presented earlier may consider elements of this innovation system
Recommendation 6 – Create a platform to discuss and monitor the competitiveness dimension of the transport programme
The increasing EU focus on the global competitiveness of its transport industry should be translated into a consistent and transparent strategy. On the one hand, competitiveness should be put at the service of a global sustainable transport paradigm that is not yet explicit enough in EU policy documents. On the other hand, global competitiveness should be defined with a consistent path, in order to provide guidance on how concrete research topics and projects may contribute to European competitiveness, rather than to protect the technological advantages of particular players within Europe. Accordingly, it would be worthwhile to provide an adequate discussion platform to discuss and monitor the competitiveness dimension of the transport programme in Horizon 2020. This approach would require improved monitoring of exploitation activities and use of results from research projects compared to those available in FP7. As this did not seem to be a priority for many project coordinators, the implementation of this monitoring should be encouraged by the EC services at least during the first years of Horizon 2020.
Recommendation 7 – Maintain the focus of international cooperation projects on network promotions
Typically, international engagement in projects has been secured by organisations joining as experts rather than as consortium partners. In some cases there is some disappointment with the participation of non-European partners.
Lack of travel budget, associated paper work and short-term projects were some of the reasons mentioned as constraints to international transport collaboration, during TRI-VALUE project interviews and these are all matters that future programmes could, and should, address.
In general, the focus of the international dimension of the FP7 should increasingly be on engaging with partners from countries outside Europe on equal terms and in programmes and activities of high mutual interest focusing on the creation of excellence networks, trying to replicate what European projects have already managed to achieve at EU level.
Additionally, formal transport knowledge networks of excellence in EU could serve as an effective means to better target EU resources and to establish a consistent collaboration policy at the international (global) level. A "transport research agenda for leadership" could serve as a reference for stakeholders and as a valuable means to bridge the gap between transport policy at the European and global level.
Recommendation 8 - Increase engagement of Civil Society organisations
The analysis performed by TRI-VALUE shows that projects have generally been successful in conveying and disseminating messages beyond the scientific community, especially towards business organisations. It also shows that while the programmes remain somehow oriented towards promoting their outputs in the research community, engagement of business organisations, including SMEs has been quite successful, which seems to help, at least in some cases, improve the commercial exploitation of results.
Horizon 2020 should build from the progress on improving involvement of businesses in TRI, which has been a focus in recent years, and put in place a strategy to improve the involvement of civil society organisations, that is undertaking knowledge transfer near the citizen. This is expected to be an effective instrument to disseminate results to a larger audience, well beyond the research community, and to also bring benefits to improve links between research activities and policy formulation, as many civil society organisations are active stakeholders in policy processes. As a first step in increasing participation of civil society organisations a review of their interests and barriers for participation may be of relevance. Accordingly we propose that the strategies to enhance participation of large companies and SMEs are critically reviewed, with the view of identifying potential ideas for further engaging civil society. For example, the advantages and disadvantages of increased involvement in ETPs or of a “target” for participation of civil society organisations in the programme shall be discussed.
Another step to achieve this may be to give adequate relevance to social issues within transport research projects, which would provide an excellent basis for the involvement of civil society organisations, and for the development of better tailored dissemination activities by researchers. To a certain extent this is already underway, as Horizon 2020 aims to focus particularly on addressing societal challenges.
Furthermore, a more fluid relationship between research services and transport services is necessary to adequately identify topics that deserve research funding; otherwise, there is a mutual risk for either choosing a topic by its innovative character without much regard for policy priorities, or choosing projects with limited- if any- innovative character, as a sort of technical assistance for too narrow, short-term policy issues. While collaboration already exists, notably at the stage of preparation of work programmes, it is recommended to ensure that such collaboration within the Commission services and with the outside world is extended to cover the entire project lifecycle. This step forward is necessary for transport research to remain relevant and to provide added value at the EU level to help formulate the right questions and to stimulate and sustain fruitful and transparent public discussion within Europe. As noted in recommendation 1, this could be organised through linking with Foresight activities of the EU.
Potential Impact:
This section looks at the overall impacts of TRI-VALUE and to dissemination and exploitation of project results, which greatly influences the extent to which the expected impacts are achieved. It is structured in two parts: the first looks at the TRI-VALUE impacts while the second addresses the measures for dissemination and exploitation of results.
Impact
The objective of TRI-Value was to undertake an ex-post evaluation of the Transport Theme within the FP7 programme. This included the assessment of the implementation and management of the FP7 transport programme, as well as the identification of the achievements and impacts of research transport projects co-funded within FP7. An integrated programme or project evaluation should include the specific objectives, its economic, social and environmental impacts, and its efficiency, effectiveness and relevance, taking as a reference the role that transport research is expected to play within the EU's relevant policies during this period (2006-2013) and beyond.
Therefore, the global and sectoral EU policies constitute the basic reference for assessing the impact of the FP7 programme, reflecting the attention within Europe to integrate crucial objectives in an increasingly challenging context, progressively dominated by the global financial and economic crisis:
- Reducing the environmental impacts of mobility, and particularly GHG emissions, through the identification and implementation of innovative technological solutions and policy measures.
- Supporting the EU transport industry and its global competitiveness.
- Addressing the emerging mobility needs linked to the six “grand challenges” faced by European societies.
TRI-VALUE fully integrated this background within the evaluation framework at both the programme evaluation level and the project evaluation level. The evaluation provided insights on how the revision and fine-tuning of EU policies in the period (2006-2013) has influenced the FP7 transport programme, and how projects have reacted to the increasing attention within topic descriptions to the emerging global environmental, societal and economic challenges. By doing so, TRI-VALUE provided information on how transport research reacts to strategic changes in European policy, and through which mechanisms, which can be particularly useful for the future design and deployment of Horizon 2020.
Furthermore, the influence of EU policy on the contents of the transport research programme and on its projects' results operate through the action of the various agents active in the R&I, including individual researchers and research institutions, the wider R&I community (including the users of transport research results, and most particularly the EU transport industry) and the European, national, and local transport policy-makers and decision-takers (European policy community). TRI-Value explored, through the review at the programme and project levels, the evolving roles of these key actors, and made an effort to actively include them in the evaluation process through a variety of participation and dissemination tools.
As noted at proposal stage, the analysis of the impacts of TRI-VALUE is a difficult topic in itself. Each of the project objectives was articulated at a relatively high level of generality. Collectively, they constitute the overall sense of direction that European transport research has sought in recent years. However, they are difficult concepts against which to gather evidence of the effectiveness of individual research projects, or, indeed, even research programmes.
In the next paragraphs the impacts of TRI-VALUE are presented with greater level of detail according to the target group: the community of researchers and organizations concerned, the community of transport researchers, the Industry, the Transport Community and the Final Users, and the European Policy community. In the end a specific summary section outlines the most important impacts in terms of efficiency, effectiveness and relevance of TRI-VALUE project activities.
Impact within the community of researchers and organizations concerned with the methodology of project and programme evaluation
A carefully conceived, clearly articulated and competently collated evidence base is critical for an effective and broad impact of any ex-post evaluation exercise. Considering the limited time and resources available to undertake the ex-post assessment, TRI-VALUE project decided not to develop a new evaluation methodology, employing instead the Research Impact Pathway (RIP) model of evaluation, with some fine-tuning activities.
Once again the RIP has proved effective in providing a structure and rationale to guide the assessment, which can be communicated and understood by users and project stakeholders. The review of other research assessments performed in an initial stage of the TRI-VALUE project proved to be a useful exercise, notably supporting the introduction of the auscultation of end-users in the assessment process.
A good evaluation methodology and practice is important to many government and quasi-governmental organizations. The framework applied in TRI-VALUE is in itself a contribution to the improvement of management practices in these organizations. Moreover, several conclusions and recommendations address the need to improve the mechanisms to measure research impacts. If properly followed-up by the European Commission and other research funders they will allow for an improvement of the capabilities to measure the outputs and impacts of research programmes. Accordingly, there may be an impact on the international community of evaluation researchers and practitioners who may benefit from the positive and negative experiences and insights of the assessment performed in TRI-VALUE.
Impact within the community of transport researchers
In the proposal stage this was identified as the level at which TRI-Value would be expected to have its largest direct impact. Social progress in areas such as transport comes from the interaction of two ‘blades’ of a pair of scissors: one ‘blade’ consists of the policy objectives articulated by elected representatives (subsequently fleshed out and developed by public officials working in national and international organisations) and the other ‘blade’ consists of the knowledge that resides within the research community (about the how and the what of effective transport research). TRI-Value aimed to communicate to the international community of transport researchers on how and what impact they were having with policy makers.
A good fit between the research approach and the research problem is a major determinant of a successful outcome. With its activities, including those related with the data collection process, TRI-Value promoted a large scale reflection on the ways research was organised and its wider impacts. While it was concluded that these links are very hard to establish TRI-VALUE believes that the process of reflection performed in the framework of the project – which involved hundreds of members of the transport research community, from the researchers themselves to end-users or Commission project officers – does have an impact of putting the issue of the impacts of research in the agenda. Moreover, the results of the TRI-VALUE analysis do throw light on how to undertake research tasks in an effective way.
The TRI-VALUE review of FP7 research efforts also allowed identifying remaining and emerging knowledge gaps, which should be addressed in the future. This will have a significant impact on the transport research community, in terms of how it positions itself and develops its skill base to address emerging challenges.
Impact within the Industry, the Transport Community and the Final Users
Although a part of the transport research community discussed above, the transport industry has a particularly relevant role within the innovation cycle, and has played an increasing role within FP7. The transport industry plays different roles within transport research: as a major user of innovation, and responsible for the crucial process of pushing innovation towards market implementation, the industry has played an increasingly influential role in setting the FP7 transport research agenda, mainly through the activities of the ETPs. Furthermore, the industry has been actively involved as partner in an increasing number of research projects. Last but not least, the industry is a key target of the dissemination activities of most of the projects funded by FP7.
TRI-Value reviewed the role of the industry in FP7 as an integrated dimension of its evaluation process. This review highlighted weaknesses and strengths of the industry participation in transport FP7 research and discussed how it has contributed to the global competitiveness goals stated within the "Innovation Union" flagship initiative. These results, which were disseminated at Europe’s flagship transport research event will hopefully raise awareness on the FP7 results among EU industrial stakeholders and will serve them to better tune their strategies and approaches to Horizon 2020.
The transport community includes also operators and managers of transport infrastructure and services and their final users, as well as a variety of social and economic actors. For them, FP7 projects should have been effective in providing new evidence for a better understanding of the transport system, its complex socio-economic impacts and its influence on their respective areas of interest. Within the review process, TRI-Value assessed how these stakeholders have been reached out by project and programme activities, and provided targeted recommendations to further empower these stakeholders and to raise their interest and awareness on FP7 research results.
Impact within the European Policy community
At the broadest level, the key impact of TRI-VALUE is to underpin the effort to keep improving the role of transport research for achieving the vision of the sensitive European transport system. As the Framework Programme moves forward towards the new Horizon 2020 concept, the FP7 final review proposed by TRI-Value was able to provide a picture on how research interacts with transport policy makers and decision takers at the European, national, regional and local level. This analysis also allowed the identification of relevant areas which were less covered in FP7 and deserve further attention in Horizon 2020.
It was clear from the onset that TRI-Value will be just one component of a much broader debate at Commission and other levels. However, the analysis performed in TRI-VALUE was debated with people from different Commission departments responsible for transport research, making the project a forum for an evidence driven debate not only on the results of past transport research but also on how to improve them in the future.
Summary of TRI-VALUE Impacts
As noted above, TRI-VALUE delivered a better understanding of the functioning and impacts of transport research activities financed within the FP7. A set of four areas were assessed: implementation and management of the programme; achievements and impacts of transport research; efficiency, effectiveness and relevance of the funding; and sustainability and utility of different research programmes. In each of these areas the project produced tangible results, all contributing to the final aim of assessing how the EU’s investment in FP7 transport research has contributed to greener smarter and more integrated transport systems and to the achievement of policy goals (Europe 2020, Innovation Union and Transport Policy). We present below a summary of the impacts that each of those results are expected to generate.
Assessment of the level of achievement of programme implementation objectives, users’ satisfaction with management procedures and comparison of the performance with programmes in other economies
TRI-VALUE analysis clearly demonstrated the level of achievement of programme objectives and provided insights on how to ensure that the performance is measured and improved. TRI-VALUE work in this also provided information on how research interacts with transport policy makers and decision takers at the European, national, regional and local level. Concrete recommendations for improvement of programme management procedures were provided.
TRI-Value extended the current understanding of the theory and practice of project and programme evaluation in transport, providing an innovative methodology which allows an effective combination of quantitative and qualitative assessment
The work of TRI-VALUE clearly increased the knowledge-base on the assessment of research and innovation funding programmes. This allowed for the introduction of several improvements in the Research Impact Pathway approach and data collection mechanisms that allows both quantitative and qualitative assessment of research funding programmes to be undertaken simultaneously.
TRI-VALUE evaluated FP7 transport research projects and assessed the extent to which they achieved the specific objectives listed in the work programme and respective calls; the project also assessed the economic, social and environmental impacts of transport research co-financed by FP7
The analysis performed in TRI-VALUE allows for the identification of patterns of success, i.e. a better understanding of how to undertake research tasks in an effective way. Moreover the project identified remaining and emerging knowledge gaps, which should be addressed in the future. The project provided concrete results of research funding and assessed positive and negative outcomes as well as a qualitative and quantitative assessment of impacts on excellent science, industrial leadership and societal challenges.
Measurement of the achievement of specific objectives listed in the work programme or calls and of the contribution to achieve major EU policies, taking into account the investment in transport research and innovation; Assessment the value added of research at EU level and comparison with other regional, national or international innovation policy landscapes in large and countries
TRI-VALUE provided a picture on how research interacts with transport policy makers and decision takers at the European, national, regional and local level and promoted a better understanding on how research results have influenced decision takers and policy makers in formulating their transport visions.
Results also raise attention to areas of new concern towards which future transport research efforts could be well advised to focus attention.
TRI-VALUE outlined patterns of collaboration with research programmes in other economies and provides concrete recommendations for improvement of transport research funding programmes. As a result it is an effort to keep improving the role of transport research for achieving the vision of the sensitive European transport system proposed by the EU policy documents.
Dissemination and Exploitation of Results
As TRI-Value is an ex-post evaluation type of project, and therefore it is focused mainly on activities related to data collection, assessment and analysis of results and development of conclusions and recommendations, the dissemination strategy has been based on the activities that: (i) efficiently engages participation of relevant stakeholders to the project activities (i.e. data collection consultation meetings and events), (ii) creates awareness about the objectives of the project and the upcoming activities, and (iii) informs interested parties about the main findings and outcomes of the project.
Therefore, the main dissemination activities implemented during the TRI-Value project were the following:
- Identification, profiling and engagement of stakeholders (i.e. ETP - European Technological Platforms, representatives of the EC (Directorates-General), Participants from FP7 projects (coordinators and partners) and Other stakeholders (Universities, industry, etc.) through direct contact, or intermediaries (e.g. ETP secretariats, PO);
- Attendance to Civitas Forum and awareness concerning the realisation of questionnaire to coordinators
- Presence in TRIP Portal – dissemination of project´s objectives and activities (e.g. questionnaires);
- Presence in ETP websites;
- Creation of LinkedIn group, for broad dissemination of project activities, results and announcements on relevant events, etc.;
- Participation in a session of TRA2014 (Paris) to present the main findings of the project.
- Presence in the Inforday for Horizon 2020
- Organisation of 2 Workshop with EC and relevant stakeholders to present the main findings and gather feedback;
- Production of relevant visual identity (logo, PPT and word templates) to be used when necessary.
The exploitation of TRI-VALUE results will be fourfold.
First, the European Commission services are expected to use the information made available by the project as an input to a wider ex-post assessment of the FP7, thus helping the Commission to comply with its legal obligations towards the European Parliament and Council. Second, the European Commission services are also expected to make use of the conclusions and recommendations of TRI-VALUE to improve transport research and innovation in the new framework programme, i.e. Horizon 2020. Third, the research community, in particular the ETPs, has in TRI-VALUE a basis for reflecting on their participation in transport research programme, and thus can use the results of TRI-VALUE as a starting point for identifying areas for improvement and selecting appropriate measures to be undertaken. Fourth, the research community as a whole, but in particular consortium members, may find in TRI-VALUE a methodology for assessment of research instruments which may be transferable to other areas. Academic-oriented partners may use this information for publications and/or teaching materials and consultancies may exploit this methodology to position themselves as research programmes assessment service providers, which may bring additional business to the participant consultancies (in particular the SMEs).
List of Websites:
Project Website (LinkedIn Group): http://goo.gl/Tbn4kg
Coordinator Contact Details:
Ms. Daniela Carvalho, daniela.carvalho@tis.pt
TRANSPORTES, INOVAÇÃO E SISTEMAS, S.A.
AV. MARQUÊS DE TOMAR, 35, 6ºDRT, 1050-153 LISBOA, PORTUGAL
T: +351 213 504 400
www.tis.pt