Periodic Reporting for period 1 - TIER2 (TIER2: ENHANCING TRUST, INTEGRITY AND EFFICIENCY IN RESEARCH THROUGH NEXT-LEVEL REPRODUCIBILITY IMPACT PATHWAYS)
Periodo di rendicontazione: 2023-01-01 al 2023-12-31
TIER2 is a three year, EC-funded project to investigate the causes, consequences and possible solutions of perceived poor levels of reproductivity of research. Reproducibility, roughly defined as the possibility for the scientific community to obtain the same results as the originators of specific findings, is often claimed as a central principle of scientific methods in many disciplines. Recently, concerns about a “reproducibility crisis” have grown in a variety of disciplines, especially in behavioural and medical sciences. This has been exacerbated by key problems such as a lack of transparency in reporting, data, and analysis, lack of replication studies, publication bias towards reporting of positive results, and a growing awareness of questionable research practices. Poor levels of reproducibility are seen as a serious threat to scientific self-correction, efficiency of research processes, and societal trust in research results.
Yet “reproducibility” is not an easily defined concept. Multiple understandings exist (including a labyrinth of competing definitions), its relevance as an indicator of research quality differs greatly based on the nature of the research, and there are also a host of socio-technical factors at play (for example, access to tools and infrastructures, incentive systems, research cultures).
In response to these challenges, TIER2 centres epistemic diversity (i.e. diverse approaches to how knowledge is obtained) by selecting three broad research areas - social, life, and computer sciences, and two cross-disciplinary stakeholder groups - research publishers and funders, to systematically investigate reproducibility across contexts. Through coordinated co-creation with these communities, TIER2:
- Examines the epistemological, social, and technical factors that shape reproducibility across contexts
- Builds a state-of-the-art evidence-base on extent and efficacy of existing reproducibility interventions and practices
- Uses co-creation techniques including scenario-planning, backcasting, and user-centred design to conceive and co-design new reproducibility-related tools and practices with our stakeholder groups
- Implements and assesses these new tools via a series of pilot activities
Through these activities, TIER2 aims to increase knowledge and awareness, build capacity, and contribute innovative new solutions to increase reproducibility in ways sensitive to the rich variety of research cultures, methods and results, Thereby, we aim to ultimately contribute to increasing the re-use and overall quality of research results and consequently boost trust, integrity and efficiency in research.
Yet “reproducibility” is not an easily defined concept. Multiple understandings exist (including a labyrinth of competing definitions), its relevance as an indicator of research quality differs greatly based on the nature of the research, and there are also a host of socio-technical factors at play (for example, access to tools and infrastructures, incentive systems, research cultures).
In response to these challenges, TIER2 centres epistemic diversity (i.e. diverse approaches to how knowledge is obtained) by selecting three broad research areas - social, life, and computer sciences, and two cross-disciplinary stakeholder groups - research publishers and funders, to systematically investigate reproducibility across contexts. Through coordinated co-creation with these communities, TIER2:
- Examines the epistemological, social, and technical factors that shape reproducibility across contexts
- Builds a state-of-the-art evidence-base on extent and efficacy of existing reproducibility interventions and practices
- Uses co-creation techniques including scenario-planning, backcasting, and user-centred design to conceive and co-design new reproducibility-related tools and practices with our stakeholder groups
- Implements and assesses these new tools via a series of pilot activities
Through these activities, TIER2 aims to increase knowledge and awareness, build capacity, and contribute innovative new solutions to increase reproducibility in ways sensitive to the rich variety of research cultures, methods and results, Thereby, we aim to ultimately contribute to increasing the re-use and overall quality of research results and consequently boost trust, integrity and efficiency in research.
During the first year of TIER2, the partners have conducted a broad range of activities which lay the groundwork for the rest of the project, and for increased reproducibility across contexts.
- The consortium launched the “TIER2 Award for Establishing a Reproducibility Network” in countries which the EC targets to widen participation from across Europe and beyond. The aim of Reproducibility Networks is to foster rigorous research practices and increase the trustworthiness of scientific work. Two consortia (one from Georgia, one from Ukraine) were awarded €5,000 each to establish a reproducibility network in their country.
- The consortium worked on building the conceptual basis for the rest of the project. Through seven distinct and ambitious scientific studies, we explored definitions, meanings, and conditions of and for reproducibility, as well as on how changes in behaviour might come abou
- As part of the conceptual work, our partners paired up with researchers from the OSIRIS project, conducting a comprehensive review of the existing evidence on the efficacy of interventions aimed at increasing reproducibility.
- Co-creation is at the centre of TIER2. In a series of workshops, invited experts imagined potential future pathways for reproducibility, deducing actionable steps necessary to arrive at the preferred future scenarios.
- To foster our engagement with the community, we held workshops with representatives from funders and publishers. These provided invaluable information which informs the development of tools and practices aimed at making reproducible research practice easier and more widely adopted and secures their engagement for the remainder of TIER2.
- Finally, our partners developed plans for eight “pilot” activities, which will develop and test specific tools and interventions aimed at raising current levels of reproducibility.
- The consortium launched the “TIER2 Award for Establishing a Reproducibility Network” in countries which the EC targets to widen participation from across Europe and beyond. The aim of Reproducibility Networks is to foster rigorous research practices and increase the trustworthiness of scientific work. Two consortia (one from Georgia, one from Ukraine) were awarded €5,000 each to establish a reproducibility network in their country.
- The consortium worked on building the conceptual basis for the rest of the project. Through seven distinct and ambitious scientific studies, we explored definitions, meanings, and conditions of and for reproducibility, as well as on how changes in behaviour might come abou
- As part of the conceptual work, our partners paired up with researchers from the OSIRIS project, conducting a comprehensive review of the existing evidence on the efficacy of interventions aimed at increasing reproducibility.
- Co-creation is at the centre of TIER2. In a series of workshops, invited experts imagined potential future pathways for reproducibility, deducing actionable steps necessary to arrive at the preferred future scenarios.
- To foster our engagement with the community, we held workshops with representatives from funders and publishers. These provided invaluable information which informs the development of tools and practices aimed at making reproducible research practice easier and more widely adopted and secures their engagement for the remainder of TIER2.
- Finally, our partners developed plans for eight “pilot” activities, which will develop and test specific tools and interventions aimed at raising current levels of reproducibility.
Our conceptual and empirical work on reproducibility has delivered the following results:
- In our conceptual work (Task 3.1) we established two key factors for reproducibility: relevance and feasibility. Investigating to which degree reproducibility concerns are relevant and feasible for a given research approach will enable tailoring policy towards different ways of producing knowledge. In turn, this will lead to better and more appropriate uptake of measures to boost reproducibility. In our evidence synthesis work (Task 3.2) we have worked to synthesise knowledge on tools and practices to improve reproducibility across scholarly disciplines and contexts. This work is currently under finalisation for sharing with the community. Once complete, it will advance the state-of-the-art by giving researchers, funders, publishers and others effective information on what works, in which circumstances, to increase reproducibility, as well as identify where evidence is currently lacking.
- In our work aiming to establish potential future pathways for reproducibility (Task 4.1) our participants emphasized the need for a cultural shift towards quality over quantity, with reproducibility becoming the norm. This involves developing common understandings of definitions across disciplines, implementing incentives for reproducible practices, and developing clear guidelines and infrastructures to foster reproducibility.
- In our conceptual work (Task 3.1) we established two key factors for reproducibility: relevance and feasibility. Investigating to which degree reproducibility concerns are relevant and feasible for a given research approach will enable tailoring policy towards different ways of producing knowledge. In turn, this will lead to better and more appropriate uptake of measures to boost reproducibility. In our evidence synthesis work (Task 3.2) we have worked to synthesise knowledge on tools and practices to improve reproducibility across scholarly disciplines and contexts. This work is currently under finalisation for sharing with the community. Once complete, it will advance the state-of-the-art by giving researchers, funders, publishers and others effective information on what works, in which circumstances, to increase reproducibility, as well as identify where evidence is currently lacking.
- In our work aiming to establish potential future pathways for reproducibility (Task 4.1) our participants emphasized the need for a cultural shift towards quality over quantity, with reproducibility becoming the norm. This involves developing common understandings of definitions across disciplines, implementing incentives for reproducible practices, and developing clear guidelines and infrastructures to foster reproducibility.