Periodic Reporting for period 2 - TIER2 (TIER2: ENHANCING TRUST, INTEGRITY AND EFFICIENCY IN RESEARCH THROUGH NEXT-LEVEL REPRODUCIBILITY IMPACT PATHWAYS)
Période du rapport: 2024-01-01 au 2025-12-31
TIER2 was a three year, EC-funded project (2023-2025) to investigate the causes, consequences and possible solutions of perceived poor levels of reproductivity of research. Reproducibility, roughly defined as the possibility for the scientific community to obtain the same results as the originators of specific findings, is often claimed as a central principle of scientific methods in many disciplines. Recently, concerns about a “reproducibility crisis” have grown in a variety of disciplines, especially in behavioural and medical sciences. This has been exacerbated by key problems such as a lack of transparency in reporting, data, and analysis, lack of replication studies, publication bias towards reporting of positive results, and a growing awareness of questionable research practices. Poor levels of reproducibility are seen as a serious threat to scientific self-correction, efficiency of research processes, and societal trust in research results.
Yet “reproducibility” is not an easily defined concept. Multiple understandings exist (including a labyrinth of competing definitions), its relevance as an indicator of research quality differs greatly based on the nature of the research, and there are also a host of socio-technical factors at play (for example, access to tools and infrastructures, incentive systems, research cultures).
In response to these challenges, TIER2 centred epistemic diversity (i.e. diverse approaches to how knowledge is obtained) by selecting three broad research areas - social, life, and computer sciences, and two cross-disciplinary stakeholder groups - research publishers and funders, to systematically investigate reproducibility across contexts. Through coordinated co-creation with these communities, TIER2:
- Examined the epistemological, social, and technical factors that shape reproducibility across contexts
- Built a state-of-the-art evidence-base on extent and efficacy of existing reproducibility interventions and practices
- Used co-creation techniques including scenario-planning, backcasting, and user-centred design to conceive and co-design new reproducibility-related tools and practices with our stakeholder groups
- Implemented and assessed these new tools via a series of pilot activities
- Conducted a series of capacity-building activities including supporting new Reproducibility Networks, creating training modules, a developing a ReproducibilityHub
Yet “reproducibility” is not an easily defined concept. Multiple understandings exist (including a labyrinth of competing definitions), its relevance as an indicator of research quality differs greatly based on the nature of the research, and there are also a host of socio-technical factors at play (for example, access to tools and infrastructures, incentive systems, research cultures).
In response to these challenges, TIER2 centred epistemic diversity (i.e. diverse approaches to how knowledge is obtained) by selecting three broad research areas - social, life, and computer sciences, and two cross-disciplinary stakeholder groups - research publishers and funders, to systematically investigate reproducibility across contexts. Through coordinated co-creation with these communities, TIER2:
- Examined the epistemological, social, and technical factors that shape reproducibility across contexts
- Built a state-of-the-art evidence-base on extent and efficacy of existing reproducibility interventions and practices
- Used co-creation techniques including scenario-planning, backcasting, and user-centred design to conceive and co-design new reproducibility-related tools and practices with our stakeholder groups
- Implemented and assessed these new tools via a series of pilot activities
- Conducted a series of capacity-building activities including supporting new Reproducibility Networks, creating training modules, a developing a ReproducibilityHub
TIER2 partners conducted a broad range of activities which lay the groundwork for increased reproducibility across contexts far beyond the project. Our work and achievements include:
- Conceptual work to better define reproducibility and develop a framework for the assessment of the relevance and feasibility of reproducibility across epistemic contexts, detailed in two academic publications
- Evidence synthesis work to bring together current knowledge on the efficacy of reproducibility interventions across disciplines (over 36,000 articles screened), as well as to investigate reproducibility issues in novel epistemic contexts (AI and qualitative research)
- Piloting and experimentation of a series of eight innovative tools and practices for reproducibility relevant to a range of stakeholders (including researchers in different disciplines, funders, publishers) and covering various stages of the research lifecycle (incl. policy planning, research planning, research workflows and checklists. publication and monitoring)
- A range of capacity-building activities related to development of networks, training, and reproducibility resources
- A policy roadmap including final recommendations for a range of stakeholders
- All this was supported by a range co-creation activities to include stakeholder engagement and input at every stage.
- Conceptual work to better define reproducibility and develop a framework for the assessment of the relevance and feasibility of reproducibility across epistemic contexts, detailed in two academic publications
- Evidence synthesis work to bring together current knowledge on the efficacy of reproducibility interventions across disciplines (over 36,000 articles screened), as well as to investigate reproducibility issues in novel epistemic contexts (AI and qualitative research)
- Piloting and experimentation of a series of eight innovative tools and practices for reproducibility relevant to a range of stakeholders (including researchers in different disciplines, funders, publishers) and covering various stages of the research lifecycle (incl. policy planning, research planning, research workflows and checklists. publication and monitoring)
- A range of capacity-building activities related to development of networks, training, and reproducibility resources
- A policy roadmap including final recommendations for a range of stakeholders
- All this was supported by a range co-creation activities to include stakeholder engagement and input at every stage.
TIER2 has delivered:
1. Enhanced understanding: Delivered through both a new Conceptual framework for evaluating the relevance and feasibility of reproducibility as a research practice and criterion of research quality in diverse settings, and Evidence synthesis to create a knowledge base on the efficacy of reproducibility interventions.
2. Innovative tools and practices: Delivered through development, implementation, and evaluation of eight highly co-creative pilot activities to create new reproducibility-related tools and practices focused on the social, life, and computer sciences, as well as research publishers and funders. Pilots covered topics from planning tools and reproducible computational workflows to editorial checks, dashboard-based monitoring, and funder-level promotion plans. Our outcomes emphasise the importance of early-stage planning, intuitive infrastructures, cross-stakeholder action and the essential role of research communities.
3. Increased capacity: Delivered through the new Reproducibility Hub of resources on the Embassy of Good Science platform, an expansive training course hosted by the OpenPlato platform, creation of three new Reproducibility Networks in Widening Participation countries, and creation of a new platform for meta-research collaboration (MERRI). These activities strengthened skills, awareness, and community infrastructures across Europe
4. Policy roadmap: Our co-created final recommendations outline priorities for improving research reproducibility, addressed to researchers, publishers, funders and institutions in Europe and beyond. The recommendations emphasise the need for context-sensitive norms, policymaking that accounts for epistemic diversity across research fields, as well as highlighting the need for stronger incentives, open practices, and dedicated support to enhance the transparency and trustworthiness of AI research.
An approachable overview of all these results is available in the Deliverable report ‘D3.2 Validated key impact pathways for reproducibility, including recommendations’: https://osf.io/bzadk/files/f56us(s’ouvre dans une nouvelle fenêtre)
1. Enhanced understanding: Delivered through both a new Conceptual framework for evaluating the relevance and feasibility of reproducibility as a research practice and criterion of research quality in diverse settings, and Evidence synthesis to create a knowledge base on the efficacy of reproducibility interventions.
2. Innovative tools and practices: Delivered through development, implementation, and evaluation of eight highly co-creative pilot activities to create new reproducibility-related tools and practices focused on the social, life, and computer sciences, as well as research publishers and funders. Pilots covered topics from planning tools and reproducible computational workflows to editorial checks, dashboard-based monitoring, and funder-level promotion plans. Our outcomes emphasise the importance of early-stage planning, intuitive infrastructures, cross-stakeholder action and the essential role of research communities.
3. Increased capacity: Delivered through the new Reproducibility Hub of resources on the Embassy of Good Science platform, an expansive training course hosted by the OpenPlato platform, creation of three new Reproducibility Networks in Widening Participation countries, and creation of a new platform for meta-research collaboration (MERRI). These activities strengthened skills, awareness, and community infrastructures across Europe
4. Policy roadmap: Our co-created final recommendations outline priorities for improving research reproducibility, addressed to researchers, publishers, funders and institutions in Europe and beyond. The recommendations emphasise the need for context-sensitive norms, policymaking that accounts for epistemic diversity across research fields, as well as highlighting the need for stronger incentives, open practices, and dedicated support to enhance the transparency and trustworthiness of AI research.
An approachable overview of all these results is available in the Deliverable report ‘D3.2 Validated key impact pathways for reproducibility, including recommendations’: https://osf.io/bzadk/files/f56us(s’ouvre dans une nouvelle fenêtre)