Transparency instruments to quantify the method transparency, analytic robustness, and replicability of empirical research

Project Information

TransparencyMeters

Grant agreement ID: 793669

DOI

10.3030/793669

Project closed

EC signature date 19 March 2018

Start date 1 September 2018

End date 31 August 2020

Funded under

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Total cost

€ 160 800,00

EU contribution

€ 160 800,00

Coordinated by

KATHOLIEKE UNIVERSITEIT LEUVEN
Belgium

Periodic Reporting for period 1 - TransparencyMeters (Transparency instruments to quantify the method transparency, analytic robustness, and replicability of empirical research)

Reporting period: 2018-09-01 to 2020-08-31

Summary of the context and overall objectives of the project

A basic requirement in science is that published findings are sufficiently trustworthy. The current state-of-the-art strategy to assess the trustworthiness of findings involves meta-analyzing all known findings on a topic. Such strategy, however, is seriously flawed because it does not account for various forms of researcher biases that contaminate each study included in a meta-analysis. Consequently, it is currently not possible for researchers (and the public) to determine how much one should trust a published finding based on traditional meta-analyses. To address this problem, we propose here to develop 3 new meta-scientific instruments (or meters) to quantify the 3 most fundamental trustworthiness aspects of a study: method transparency, analytic robustness, and effect replicability. For each instrument, a corresponding metric (i.e. unit of measurement) will be developed to quantify the degree to which a published study exhibits method transparency, analytic robustness, and effect replicability. This will be achieved by applying the metrics to CurateScience.org a web platform that tracks and quantifies the trustworthiness of studies in a crowdsourced, incremental fashion over time. These transparency instruments will revolutionize how meta-analyses are conducted, substantially improving the quality and validity of meta-analytic conclusions in all scientific fields. This will accelerate cumulative knowledge development and the exploitation of scientific findings to develop solutions to address Horizon’s 2020 societal challenges of European citizens. These new instruments will advance the field of meta-science, but also have the salutary effect of accelerating the uptake of open science practices among social science researchers. The proposed work is directly relevant to European policy objectives related to Open Science and Research Integrity to increase the openness, access to, and re-use of publicly-funded research and data.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

A foundational transparency compliance system was designed and developed, which includes various transparency and reproducibility standards, curation tools, and a suite of web apps that all players (researchers, journals, universities, and funding agencies) can use to ensure and track the transparency and credibility of research (see Figure 2 for a visual summary of the current achievements, state, and road map of the Curate Science transparency and credibility curation system). Achievements, outputs, and concrete deliverables include:
►Tech infrastructure fully implemented (details), which provides the foundation to add/plug in the various standards, tools, web apps, and products
►Transparency standards across 14 article types for all fields
►Article accessibility standards and funder grant review transparency standards
►Reproducibility curation (study-level/article-level, prototype) and reproducibility viewing (designs/prototype)
►Result robustness (designed)
►Curate Scholar product (early beta released v0.4.2 50 beta testers, hundreds on waiting list; open source code base; bugs/issues tracker & feature requests)
►Curate Journal (designed; prototypes: 1, 2, 3, 4, 5; interested users confirmed)
►Curate University (designed; interactive prototypes: 1, 2, 3; interested users confirmed)
►Curate Funder (designed; interactive prototypes: 1, 2, 3, 4; interested users confirmed)
►Curated content, as part of general crowdsourced curation platform (~500 articles, ~4K replications (filter by “Replication”); articles among most transparent in the history of modern science)

We disseminated our project results and progress via a published paper, our public website (https://CurateScience.org) our platform’s newsletter (http://us8.campaign-archive2.com/home/?u=0833383918fc50773891d363a&id=aaad5734e3) Twitter (via Curate Science’s Twitter account: https://twitter.com/curatescience and the fellow’s Twitter account: https://twitter.com/eplebel) and a YouTube video podcast (Saving Science Show: https://www.youtube.com/playlist?list=PL2Hr1_liJSKsGKPV8XHlSfr50YysaVBcH) the latter which was an additional deliverable.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The foundational tech infrastructure and the suite of transparency standards, tools, and apps that we have delivered (details) will enhance innovation and benefit society in several ways. As part of the Curate Scholar product, researchers can curate the transparency of their publications, which also helps transform the research culture in academia where being transparent is the new default (rather than closed by default, and then open up later). Our flexible transparency standards, and broader integrated transparency compliance system, can also be used by journals, universities, and funding agencies, to ensure that all submitted articles, employees/professors, and grantees, respectively, meet a minimum transparency level of their choosing. Buy-in from players is enhanced given the integrated system, which saves time and minimize the bureaucracy of ensuring transparency compliance (e.g. a researcher only needs to meet transparency requirements once per article, for e.g. at a journal, which would mean that their article would automatically be in compliance with their university and funding agency). Minimum transparency standards will also revolutionize systematic reviews and meta-analyses whereby only studies meeting a specific transparency standard would be eligible for inclusion (and/or one could examine whether effect sizes shrink in more transparently reported studies). Of note, our tools and broader platform are fully aligned with the open science goals as covered in the EU’s policy objectives as part of the Horizon 2020 programme.

cs-state-of-and-roadmap-1.png