Periodic Reporting for period 1 - OSIRIS (Open Science to Increase Reproducibility in Science)
Okres sprawozdawczy: 2023-01-01 do 2023-12-31
Reproducibility is crucial to progress and impact of Research and Innovation (R&I) as it confirms or corrects the outcomes of single studies, resulting in higher quality research, more reliable and implementable outcomes, and reduction of research costs.
Embedding reproducibility in the strategy and design of research should thus be regarded as a key precondition to research quality. Unfortunately, only a fraction of published studies can be reproduced due to e.g science’s need to be continuously innovative, pressure to publish, a lack of transparent reporting, and career assessments based on quantity rather than quality.
Currently, the burden of improving reproducibility lies with the researchers, as there are limited to no incentives to encourage active engagement in reproducibility and ideas for improvement have never been tested in practice.
We need a paradigm and culture shift to reform the R&I system both from the top down and the bottom up to regain overall trust in science. Open Science to Increase Reproducibility In Science (OSIRIS) aims to facilitate this shift by systematically gathering knowledge on the underlying drivers, testing effective evidence-based solutions, identifying incentives for reproducibility by stakeholders, and embedding reproducibility in research design.
The ambition of OSIRIS is to reform the R&I system such that reproducibility is more accepted, practiced, and recognized within global scientific practice by 2026.
Objectives:
1. To understand the underlying drivers and effective interventions that increase reproducibility at funding, publishing, university, and researcher-level using systematic literature review, evidence mapping, policy audits, and interviews and focus group discussions with
stakeholders. Results will be distributed through an open knowledge base and Open Access (OA) publications to optimally reach global academia.
2. To develop and test effective, evidence-based solutions for the reproducibility crisis across various stakeholders in policy and research practice by utilising well-controlled Randomised Controlled Trials (RCTs) rather than mere pilots, develop dashboards of indicators of
reproducible research practices, and providing funders, publishers, researchers, and peer reviewers with guidance for judging reproducibility.
3. To embed reproducibility in the strategy and design of research projects by informing researchers and convincing funders and journals to include measures and preconditions on reproducibility in their assessment of project proposals and articles.
4. To create a collaborative community of stakeholders that will aid in educating and implementing better reproducible research practice using our results to create guidelines and training on how researchers can embed reproducibility in the design of their research and
disseminate these widely, thereby increase the reproducibility of their scientific research. Additionally, we will perform quality audits at project and output level to test these novel practices.
Our interdisciplinary consortium is uniquely positioned to tackle these objectives. OSIRIS builds on the consortium's collective and unique knowledge and experience in OS and reproducibility, empirical research, developing and validating RCT interventions, translating findings into effective solutions, and developing inclusive stakeholder communities at different applicability levels. By combining the expertise of the consortium members, we can create an evidence-based, usable, and widely-supported package of interventions to promote reproducibility. Together with our extensive network of reproducibility leaders, journals, funders, and policy officers, many of which are represented in our AB, we will ensure uptake of our results in a wide variety of scientific fields and institutions.
Embedding reproducibility in the strategy and design of research should thus be regarded as a key precondition to research quality. Unfortunately, only a fraction of published studies can be reproduced due to e.g science’s need to be continuously innovative, pressure to publish, a lack of transparent reporting, and career assessments based on quantity rather than quality.
Currently, the burden of improving reproducibility lies with the researchers, as there are limited to no incentives to encourage active engagement in reproducibility and ideas for improvement have never been tested in practice.
We need a paradigm and culture shift to reform the R&I system both from the top down and the bottom up to regain overall trust in science. Open Science to Increase Reproducibility In Science (OSIRIS) aims to facilitate this shift by systematically gathering knowledge on the underlying drivers, testing effective evidence-based solutions, identifying incentives for reproducibility by stakeholders, and embedding reproducibility in research design.
The ambition of OSIRIS is to reform the R&I system such that reproducibility is more accepted, practiced, and recognized within global scientific practice by 2026.
Objectives:
1. To understand the underlying drivers and effective interventions that increase reproducibility at funding, publishing, university, and researcher-level using systematic literature review, evidence mapping, policy audits, and interviews and focus group discussions with
stakeholders. Results will be distributed through an open knowledge base and Open Access (OA) publications to optimally reach global academia.
2. To develop and test effective, evidence-based solutions for the reproducibility crisis across various stakeholders in policy and research practice by utilising well-controlled Randomised Controlled Trials (RCTs) rather than mere pilots, develop dashboards of indicators of
reproducible research practices, and providing funders, publishers, researchers, and peer reviewers with guidance for judging reproducibility.
3. To embed reproducibility in the strategy and design of research projects by informing researchers and convincing funders and journals to include measures and preconditions on reproducibility in their assessment of project proposals and articles.
4. To create a collaborative community of stakeholders that will aid in educating and implementing better reproducible research practice using our results to create guidelines and training on how researchers can embed reproducibility in the design of their research and
disseminate these widely, thereby increase the reproducibility of their scientific research. Additionally, we will perform quality audits at project and output level to test these novel practices.
Our interdisciplinary consortium is uniquely positioned to tackle these objectives. OSIRIS builds on the consortium's collective and unique knowledge and experience in OS and reproducibility, empirical research, developing and validating RCT interventions, translating findings into effective solutions, and developing inclusive stakeholder communities at different applicability levels. By combining the expertise of the consortium members, we can create an evidence-based, usable, and widely-supported package of interventions to promote reproducibility. Together with our extensive network of reproducibility leaders, journals, funders, and policy officers, many of which are represented in our AB, we will ensure uptake of our results in a wide variety of scientific fields and institutions.
Towards the achievement of these goals, in the first 12 months of the project we have completed the following deliverables:
WP1: Project Handbook (M3)
WP1: Data management plan (M6)
WP6: Dissemination and Communication Plan (M6)
WP6: Project website (M2)
WP6: Open Science plan (M6)
WP6: RP1 Updated Dissemination and Communication Plan (M12)
WP6: Policy Briefing RP1 (M13)
-> The systematic scoping review have identified 88 papers that provide evidence for reproducibility interventions and additionally 138 papers summarizing the views of multiple researchers about barriers or drivers of reproducibility.
-> Interviews with researchers across European research institutions are ongoing to investigate what reproducibility means in research. Additionally, the outcome of the interviews will allow us to investigate what reproducibility means in research and how different researchers personally view it; to elicit which experiences researchers have regarding reproducibility in their own research, within their field and in science in general; to examine how researchers execute reproducibility in practice; and to understand drivers, barriers and facilitators for reproducibility in the researcher’s context.
-> We have determined a preliminary list of the stakeholders groups to target in the focus group discussions, to further explore the findings of scoping review and interviews.
-> The development of a protocol to audit policies and guidelines of institutions, funders and publishers for reproducibility measures is ongoing.
-> Initiate the development of a specific checklist for reproducibility.
-> Initiate the setting up of the OSIRIS researchers' network for reproducibility checks.
-> Made an inventory of already existing automatic systems for reproducibility checking, and discussed what outcome measures are relevant to measure when one aims to measure reproducibility.
Additionally, in WP6 we have created an interactive, visually pleasing, and graphical website that describes and highlights the activities carried out in the project. We mapped out, engaged and maintained a strong stakeholder network through a series of online and face-to-face events and developed an exploitation roadmap towards further validation and sustainability of OSIRIS.
WP1: Project Handbook (M3)
WP1: Data management plan (M6)
WP6: Dissemination and Communication Plan (M6)
WP6: Project website (M2)
WP6: Open Science plan (M6)
WP6: RP1 Updated Dissemination and Communication Plan (M12)
WP6: Policy Briefing RP1 (M13)
-> The systematic scoping review have identified 88 papers that provide evidence for reproducibility interventions and additionally 138 papers summarizing the views of multiple researchers about barriers or drivers of reproducibility.
-> Interviews with researchers across European research institutions are ongoing to investigate what reproducibility means in research. Additionally, the outcome of the interviews will allow us to investigate what reproducibility means in research and how different researchers personally view it; to elicit which experiences researchers have regarding reproducibility in their own research, within their field and in science in general; to examine how researchers execute reproducibility in practice; and to understand drivers, barriers and facilitators for reproducibility in the researcher’s context.
-> We have determined a preliminary list of the stakeholders groups to target in the focus group discussions, to further explore the findings of scoping review and interviews.
-> The development of a protocol to audit policies and guidelines of institutions, funders and publishers for reproducibility measures is ongoing.
-> Initiate the development of a specific checklist for reproducibility.
-> Initiate the setting up of the OSIRIS researchers' network for reproducibility checks.
-> Made an inventory of already existing automatic systems for reproducibility checking, and discussed what outcome measures are relevant to measure when one aims to measure reproducibility.
Additionally, in WP6 we have created an interactive, visually pleasing, and graphical website that describes and highlights the activities carried out in the project. We mapped out, engaged and maintained a strong stakeholder network through a series of online and face-to-face events and developed an exploitation roadmap towards further validation and sustainability of OSIRIS.
The urgent need to deliver evidence and effective solutions for science to be reproducible, trustworthy, and impactful is clear. Evidence-based knowledge and effective evidence-based intervention strategies are necessary to rigorously assess evidence-based practices toward reproducibility.
In the first year we have worked on analyzing the already existing scientific evidence about underlying drivers for reproducibility. Recently, we have started the interviews with researchers from European research institutions to gain insights from the expertise of a variety of stakeholders on drivers, motivations, barriers, and practices. The next goal is then translate these into effective solutions for increasing reproducibility at institutional, journal and funder level, and develop a collaborative and inclusive community among stakeholders. With this approach, we will be able to learn what are the barriers and motivations of the researchers and plan for effective interventions.
In the first year we have worked on analyzing the already existing scientific evidence about underlying drivers for reproducibility. Recently, we have started the interviews with researchers from European research institutions to gain insights from the expertise of a variety of stakeholders on drivers, motivations, barriers, and practices. The next goal is then translate these into effective solutions for increasing reproducibility at institutional, journal and funder level, and develop a collaborative and inclusive community among stakeholders. With this approach, we will be able to learn what are the barriers and motivations of the researchers and plan for effective interventions.