Periodic Reporting for period 1 - OPUS (Open Universal Science)
Okres sprawozdawczy: 2022-09-01 do 2023-08-31
The Open and Universal Science (OPUS) project develops coordination and support measures to reform the assessment of research and researchers at Research Performing Organisations (RPOs) and Research Funding Organisations (RFOs) towards a system that incentivises and rewards researchers to take up Open Science practices. The project is focused on achieving six main objectives to steer all project activities:
1. Conduct a comprehensive state-of-the-art on literature and initiatives for Open Science
2. Develop comprehensive interventions to implement Open Science at RPOs and RFOs
3. Develop realistic indicators/metrics to monitor/drive Open Science at RPOs and RFOs
4. Test the interventions and indicators/metrics via action plans in pilots at RPOs and RFOs
5. Use a stakeholder-driven feed-back loop to develop, monitor, refine, and validate actions
6. Synthesise outcomes into policy briefs and a revised model for researcher assessment
The project will produce several key exploitable results to share with the research and innovation community:
- State-of-the-Art on Open Science Literature and Initiatives
- Indicators/Metrics and Interventions for Researcher Assessment
- Action Plans and Mutual Learning Exercise to Implement Pilots
- Revised Open Science Career Assessment Matrix (OS-CAM2)
1. Conduct a comprehensive state-of-the-art on literature and initiatives for Open Science
2. Develop comprehensive interventions to implement Open Science at RPOs and RFOs
3. Develop realistic indicators/metrics to monitor/drive Open Science at RPOs and RFOs
4. Test the interventions and indicators/metrics via action plans in pilots at RPOs and RFOs
5. Use a stakeholder-driven feed-back loop to develop, monitor, refine, and validate actions
6. Synthesise outcomes into policy briefs and a revised model for researcher assessment
The project will produce several key exploitable results to share with the research and innovation community:
- State-of-the-Art on Open Science Literature and Initiatives
- Indicators/Metrics and Interventions for Researcher Assessment
- Action Plans and Mutual Learning Exercise to Implement Pilots
- Revised Open Science Career Assessment Matrix (OS-CAM2)
In the first project period, project activities began by conducting an initial state-of-the-art on OS Initiatives and on OS Literature. The first covered existing academic and policy literature related to researcher assessment and OS, with a focus on incentives and rewards, precarity of research careers, gender equality, industry practices, and trust. The second looked at current initiatives to improve researcher assessment and support OS, through landscaping of framework projects, of experts, organisations, networks and schemes, and development of the OPUS Stakeholder Engagement Plan. This data was used to support development of the initial set of interventions, indicators and metrics. This took the form of the draft OPUS Research Assessment Framework (RAF). The RAF will offer a unique framework for researcher assessment by covering the full spectrum of research and non-research activities conducted by researchers, focusing on OS. It combines a quantitative and qualitative approach, providing comprehensive indicators/metrics together with interventions to support implementation at RPOs and RFOs. The RAF offers maximum flexibility to RPOs and RFOs in selecting, refining, and prioritising the indicators/metrics and interventions according to their organisational interests and needs. The RAF will be tested in 5 pilot organisations, who have run a baseline audit on selected indicators and metrics, begun developing initial action plans and a framework for mutual learning exercises. Finally, key outcomes from the work carried out so far were consolidated into an initial brief to present initial findings and to share the policy recommendations that have emerged so far.
The state of the arts on literature on incentives and rewards confirmed that the current researcher assessment system remains overly focused on bibliometrics, involving peer-reviewed publications and citations in top journals. The research community needs a more comprehensive approach of altimetric, which includes research/non-research, open/closed, and quantitative/qualitative dimensions. There are, however, many policy developments at national, European, and international levels, with examples of new frameworks (including principles, guidelines, indicators/metrics, and interventions) to reform researcher assessment. Such frameworks may be adopted and adapted to incentivise and reward OS at RPOs and RFOs.
In this context, OPUS goes beyond the state of the art with the development of the first Research Assessment Framework (RAF). The draft RAF covers two levels:
• Meta-interventions as an overarching set of interventions for anyone that wishes to implement the RAF or OSCAM framework. They are intended as a starting point for consideration and implementation at the top level of an organisation;
• Indicator Level Interventions – Each indicator defined in the framework has an individual tailored set of interventions to help their implementation.
The following ten principles should guide the development and implementation of the RAF:
1. Provide a comprehensive framework of indicators and metrics for RPOs and RFOs
2. Provide a framework which applies across countries, disciplines, and organisations
3. Provide a framework which combines both quantitative and qualitative assessment
4. Focus on the assessment of individual researchers and not teams, groups, or units
5. Cover the full spectrum of activities by researchers and not just research activities
6. Offer a generic framework which allows open and non-open activities by researchers
7. Offer a specific framework which focuses on Open Science activities by researchers
8. Distinguish process, output, and outcome indicators to capture the lifecycle of activities
9. Formulate indicators and metrics at a high level of description for universal application
10. Leave selection, refinement and prioritisation of indicators and metrics to RPOs and RFOs
The RAF should be translated into a workable tool for practical implementation by RPOs and RFOs. The translation of the RAF will be different for and across RPOs and RFOs depending on the target groups of researchers under assessment, the intrinsic nature of the assessment, and the selection and prioritisation of indicators/metrics. The translation of the RAF will also differ depending on the chosen medium of implementation for presentation towards researchers (such as via assessment questionnaires or digital interfaces or tools). The RAF should be supported with concrete interventions linked to indicators/metrics and clear guidelines explaining the RAF and the assessment procedure for researchers.
The RAF will offer a unique framework for researcher assessment by covering the full spectrum of research and non-research activities conducted by researchers, focusing on Open Science, combining a quantitative and qualitative approach, providing comprehensive indicators/metrics together with interventions to support implementation at RPOs and RFOs, and offering maximum flexibility to RPOs and RFOs in selecting, refining, and prioritising the indicators/metrics and interventions according to their organisational interests and needs.
The following next steps to develop, test, and finalise the RAF will be taken in OPUS:
- First version of the RAF will be completed and made openly available in Q4 2023
- Consultation with stakeholders on the first version of the RAF in Q4 2023-Q1 2024
- Piloting of the first version of the RAF at 3 RPOs and 2 RFOS in Q1 2024-Q2 2025
- Final version of the RAF will be completed and made openly available in Q3 2025
In this context, OPUS goes beyond the state of the art with the development of the first Research Assessment Framework (RAF). The draft RAF covers two levels:
• Meta-interventions as an overarching set of interventions for anyone that wishes to implement the RAF or OSCAM framework. They are intended as a starting point for consideration and implementation at the top level of an organisation;
• Indicator Level Interventions – Each indicator defined in the framework has an individual tailored set of interventions to help their implementation.
The following ten principles should guide the development and implementation of the RAF:
1. Provide a comprehensive framework of indicators and metrics for RPOs and RFOs
2. Provide a framework which applies across countries, disciplines, and organisations
3. Provide a framework which combines both quantitative and qualitative assessment
4. Focus on the assessment of individual researchers and not teams, groups, or units
5. Cover the full spectrum of activities by researchers and not just research activities
6. Offer a generic framework which allows open and non-open activities by researchers
7. Offer a specific framework which focuses on Open Science activities by researchers
8. Distinguish process, output, and outcome indicators to capture the lifecycle of activities
9. Formulate indicators and metrics at a high level of description for universal application
10. Leave selection, refinement and prioritisation of indicators and metrics to RPOs and RFOs
The RAF should be translated into a workable tool for practical implementation by RPOs and RFOs. The translation of the RAF will be different for and across RPOs and RFOs depending on the target groups of researchers under assessment, the intrinsic nature of the assessment, and the selection and prioritisation of indicators/metrics. The translation of the RAF will also differ depending on the chosen medium of implementation for presentation towards researchers (such as via assessment questionnaires or digital interfaces or tools). The RAF should be supported with concrete interventions linked to indicators/metrics and clear guidelines explaining the RAF and the assessment procedure for researchers.
The RAF will offer a unique framework for researcher assessment by covering the full spectrum of research and non-research activities conducted by researchers, focusing on Open Science, combining a quantitative and qualitative approach, providing comprehensive indicators/metrics together with interventions to support implementation at RPOs and RFOs, and offering maximum flexibility to RPOs and RFOs in selecting, refining, and prioritising the indicators/metrics and interventions according to their organisational interests and needs.
The following next steps to develop, test, and finalise the RAF will be taken in OPUS:
- First version of the RAF will be completed and made openly available in Q4 2023
- Consultation with stakeholders on the first version of the RAF in Q4 2023-Q1 2024
- Piloting of the first version of the RAF at 3 RPOs and 2 RFOS in Q1 2024-Q2 2025
- Final version of the RAF will be completed and made openly available in Q3 2025