Periodic Reporting for period 2 - RI Impact Pathways (Charting Impact Pathways of Investment in Research Infrastructures)
Reporting period: 2019-07-01 to 2020-06-30
The specific objectives of the RI-PATHS project were:
1) Take stock of the existing approaches for impact assessment of research infrastructures and map the current and future data gathering needs of key stakeholder groups;
2) Employing systems thinking approach develop a modular impact assessment (IA) framework that represents all major impact pathways of RIs;
3) Operationalise the IA framework by defining a set of reference indicators, providing guidance on monitoring and evaluation approaches and testing feasibility with pilot RIs.
The project developed a comprehensive framework describing the socio-economic impact of research infrastructures (RIs) and their related financial investments. The framework was developed in a modular manner adapting it to a broad range of scientific domains and types of RIs. The project outcomes will to contribute to a common approach at the EU And international level and facilitate investments in research infrastructures by funding agencies and other stakeholders.
1) Take stock of the existing approaches for impact assessment of research infrastructures and map the current and future data gathering needs of key stakeholder groups;
2) Employing systems thinking approach develop a modular impact assessment (IA) framework that represents all major impact pathways of RIs;
3) Operationalise the IA framework by defining a set of reference indicators, providing guidance on monitoring and evaluation approaches and testing feasibility with pilot RIs.
The project developed a comprehensive framework describing the socio-economic impact of research infrastructures (RIs) and their related financial investments. The framework was developed in a modular manner adapting it to a broad range of scientific domains and types of RIs. The project outcomes will to contribute to a common approach at the EU And international level and facilitate investments in research infrastructures by funding agencies and other stakeholders.
RI-PATHS project pursued three objectives:
Objective 1: Carry out a comprehensive stocktaking exercise on the existing approaches for impact assessment of RIs and map the current and future data needs of the key stakeholders.
The stocktaking exercise included a literature review of the current analytical methods used for impact assessment of research infrastructures and the design, implementation and analysis of an online survey of RIs and interviews with policy makers and experts. The literature review provided a critical assessment of the strengths and weaknesses of the existing main impact assessment methods, highlighting that no single method in its traditional formulation provides a comprehensive and satisfactory answer for the needs of assessing the socio-economic impact of RIs. It was concluded that RI-PATHS guidance should include the most promising existing approaches expanding and combining them to enable fitting analysis of a broader range of impacts. T
Objective 2: Employ a systems approach to develop a modular impact assessment framework that represents all major impact pathways of distinct types of research infrastructures.
The impact assessment framework has been developed in a consecutive manner adopting and revising the initial assumptions in the course of the project. As a first step, a working note on a RI typology was developed. The classification distinguished between RIs that carry out research (basic, use-inspired and applied research) and RIs that deliver research-oriented services. The activities included two rounds of participatory workshops with RIs to further develop the design of the IA framework. The design of the first round of workshops were more exploratory sourcing input from the RI community on the most important impact areas and impact pathways to be included in the IA framework. This work led to detailing of relevant impact areas in four impact categories: 1) Impact on Human Resources; 2) Impact on Economy; 3) Impact on Society; and 4) Impact on Policy. A second round of workshops were structured around three main themes: 1) Quantifiable impacts that can be captured through available quantitative metrics (e.g. economic analysis) ; 2) Non-quantifiable impacts the assessment of which can, however, be supported by quantitative means (e.g. social network analysis, dedicated surveys); 3) Complex network effects captured through qualitative approaches (e.g. narrative-based case studies) and exploratory methods. These participatory events contributed to the prioritisation of relevant indicators and reviewed the applicability of available analytical methods.
Objective 3: Operationalisation of the IA framework by defining a set of reference indicators, providing guidance on the most appropriate monitoring and evaluation approaches and testing its feasibility with pilot research infrastructures.
The final version of the IA framework consisting of 13 generic impact pathways, key findings forming the modularity of the framework, classification of indicators along the impact pathways, more inclusive categorisation of impact areas and lists of examples for reporting tools and applicable analytical methods. The principles of the RI-PATHS IA framework and more in-depth investigation of the methods and approaches for scoping and measuring specific impact areas were undertaken during piloting exercises with the four consortium RIs, as well as three RIs that joined voluntarily.
Objective 1: Carry out a comprehensive stocktaking exercise on the existing approaches for impact assessment of RIs and map the current and future data needs of the key stakeholders.
The stocktaking exercise included a literature review of the current analytical methods used for impact assessment of research infrastructures and the design, implementation and analysis of an online survey of RIs and interviews with policy makers and experts. The literature review provided a critical assessment of the strengths and weaknesses of the existing main impact assessment methods, highlighting that no single method in its traditional formulation provides a comprehensive and satisfactory answer for the needs of assessing the socio-economic impact of RIs. It was concluded that RI-PATHS guidance should include the most promising existing approaches expanding and combining them to enable fitting analysis of a broader range of impacts. T
Objective 2: Employ a systems approach to develop a modular impact assessment framework that represents all major impact pathways of distinct types of research infrastructures.
The impact assessment framework has been developed in a consecutive manner adopting and revising the initial assumptions in the course of the project. As a first step, a working note on a RI typology was developed. The classification distinguished between RIs that carry out research (basic, use-inspired and applied research) and RIs that deliver research-oriented services. The activities included two rounds of participatory workshops with RIs to further develop the design of the IA framework. The design of the first round of workshops were more exploratory sourcing input from the RI community on the most important impact areas and impact pathways to be included in the IA framework. This work led to detailing of relevant impact areas in four impact categories: 1) Impact on Human Resources; 2) Impact on Economy; 3) Impact on Society; and 4) Impact on Policy. A second round of workshops were structured around three main themes: 1) Quantifiable impacts that can be captured through available quantitative metrics (e.g. economic analysis) ; 2) Non-quantifiable impacts the assessment of which can, however, be supported by quantitative means (e.g. social network analysis, dedicated surveys); 3) Complex network effects captured through qualitative approaches (e.g. narrative-based case studies) and exploratory methods. These participatory events contributed to the prioritisation of relevant indicators and reviewed the applicability of available analytical methods.
Objective 3: Operationalisation of the IA framework by defining a set of reference indicators, providing guidance on the most appropriate monitoring and evaluation approaches and testing its feasibility with pilot research infrastructures.
The final version of the IA framework consisting of 13 generic impact pathways, key findings forming the modularity of the framework, classification of indicators along the impact pathways, more inclusive categorisation of impact areas and lists of examples for reporting tools and applicable analytical methods. The principles of the RI-PATHS IA framework and more in-depth investigation of the methods and approaches for scoping and measuring specific impact areas were undertaken during piloting exercises with the four consortium RIs, as well as three RIs that joined voluntarily.
The final phase of the project focused on translating all project findings and key lessons learned into a dedicated user-friendly Guidebook on Socio-Economic Impact Assessment of Research Infrastructures and a web-based Toolkit that allows users to explore the information in tailored ways – in a top-down or a bottom-up approach. In a top down approach, the user can start by answering the preliminary questions that help to clarify and orient the user towards one or more specific pathways. Users can also a priori define the impact areas which they are mostly interested to explore. A bottom-up approach starts with the analysis of prior, existing and available data, previously concluded studies and ongoing assessment exercises in order to shape pathways that can further and complement this information. Devising some of the web-based functionalities, the RI-PATHS team consulted and took up the feedback from the ERIC RI community to ensure the Toolkit meets user needs and expectations. Moreover, dedicated feedback functions have been embedded in the Toolkit to enable the RI community to suggest further updates, modifications and application examples to help improve and nuance the IA framework in the future