Periodic Reporting for period 1 - AAT (Appraising and accelerating theoretical progress in memory modification research with meta-science, Bayesian statistics, and an open-collaborative approach.)
Okres sprawozdawczy: 2020-04-20 do 2022-04-19
There is growing concern about the credibility of scientific research due to low transparency, poor design choices, selective reporting bias, and statistical misuse. Poor quality research wastes tax payer's money and can lead to ineffective or even harmful social and health interventions. The scientific ecosystem is supposed to be protected by 'self-correcting' mechanisms that weed out bad science, but it is unclear if these are operating effectively. The current project took place against this backdrop. The project took a meta-research ('research-on-research') approach to investigate the problems and potential solutions of the current credibility crisis. The specific scientific and training objectives of the project were to (1) appraise and accelerate theoretical progress in memory modification research with Bayesian statistics, collaboration, and open science; and (2) create opportunities for the research fellow to develop a strong repertoire of statistical, methodological, and transferable skills, enrich their research portfolio, and consolidate their network of collaborators in the fields of meta-research and psychology.
(1) A systematic review of 168 human reconsolidation experiments found that few studies meeting the design conditions necessary to provide a robust test of reconsolidation theory. Additionally, a subset of studies that have employed the desired 2x2 factorial design, often fall foul to a statistical fallacy in which a statistically significant different between two groups, and the absence of a statistically significant difference between two other groups, is taken as an indication that there is a significant interaction effect. Bayesian reanalyses that actually test the interaction in these contexts found that there was insufficient evidence to conclude an interaction effect, thus challenging the conclusions of the original papers.
(2) We sought to describe the nature and prevalence of statistical guidance provided to authors at 330 top-ranked journals operating across 22 scientific disciplines. We observed considerable heterogeneity across disciplines, with for example, 122/165 (74%) of assessed Health & Life Sciences journals providing statistical guidance and 0/30 (0%) of assessed Formal Sciences journals providing statistical guidance. We found that p-values and confidence intervals tended to be mentioned more often that Bayesian statistics; though when guidance on Bayesian statistics was provided, it generally indicated implicit endorsement. This study provides a stratum of empirical evidence upon which to base conceptual discussions about the role journals should play in the governance of statistical analysis.
Dissemination: Hardwicke et al. (preprint, https://doi.org/hmq6(odnośnik otworzy się w nowym oknie)). Hardwicke et al. (2022, presentation at the International Congress on Peer Review and Scientific Publication)
(3) We performed an extensive cross-disciplinary review of literature addressing the topic of preregistration. Our report outlines how preregistration can serve to reduce bias, increase transparency, and calibrating confidence in scientific claims.
Dissemination: Hardwicke & Wagenmakers (preprint, https://doi.org/f76t)(odnośnik otworzy się w nowym oknie); Hardwicke (2022 presentation at the Joint Statistical Meetings); Hardwicke (2021, presentation at Ludwig Maximilian University of Munich).
(4) We tested the feasibility of an intervention intended to detect undisclosed discrepancies during peer review and flag them to authors and editors so they could be resolved prior to publication. We found that the intervention could be successfully deployed and identified relevant parameters that could aid the preparation of a future randomized controlled trial to evaluate intervention efficacy.
Dissemination: TARG Meta-Research Group & Collaborators (2022, https://doi.org/h78p(odnośnik otworzy się w nowym oknie))
(5) We summarised the expansive literature on replication/reproducibility in psychology and provided new empirical evidence on several key issues.
Dissemination: Nosek, et al. (2022, https://doi.org/g3qf(odnośnik otworzy się w nowym oknie))
(6) Scientific studies often contain critical flaws, so it is important that the research community can criticise published research. In this study we assessed how 15 top-ranked journals in each of 22 scientific disciplines (330 journals in total) handled post-publication critique (for example 'letters to the editor). We found that just over a third (123) of journals did not accept post-publication critique, and those that did often imposed strict limits on length and time-to-submit. For example, one journal required that critiques were less than 175 words long and another journal required that critiques were submitted within two-weeks of the original study being published. We also found that journals rarely published critiques — based on an assessment of 2066 randomly sampled research articles, we estimated the prevalence of post-publication critique to be 1.9%, 95% confidence interval [1.4 2.6]. We also found that although most critiques attract a response from the original authors, those authors very rarely change their minds about the original conclusions. Overall, the study demonstrated that top-ranked journals are a serious barrier to the publication of critical discourse across scientific disciplines.
Dissemination: Hardwicke et al. (2022, https://doi.org/h828)(odnośnik otworzy się w nowym oknie); Hardwicke et al. (2022, presentation at International Congress on Peer Review and Scientific Publication); Hardwicke (2022, presentation at METRICS International Forum, Stanford University); Hardwicke (2022, presentation at Society for Improving Psychological Science Conference)
(7) We examined the trajectory of citation patterns and citation valence before and after four impactful psychology studies could not be replicated in large-scale preregistered replication projects. We observed only a slight post-replication decline in favourable citations and a small increase in unfavourable citations. This suggests that in practice, the expected ‘self-correction’ effect of replication studies can fail to materialize.
Dissemination: Hardwicke et al. (2021, https://doi.org/gwt8)(odnośnik otworzy się w nowym oknie); Hardwicke (2022, presentation at Berkeley Initiative for Transparency in the Social Sciences Annual Meeting)
(8) We observed that of 140 papers retracted in psychology due to data fabrication, scientific misconduct, or error, after retraction, 88 (63%) of these papers received at least one positive citation (median = 2, interquartile range = 4, min = 0, max = 89 positive citations). This suggests that interventions are needed to improve researchers’ awareness of retractions (e.g. notifying them through their paper management software).
Dissemination: Fernández et al. (preprint, https://doi.org/hcht(odnośnik otworzy się w nowym oknie))
(2) We sought to describe the nature and prevalence of statistical guidance provided to authors at 330 top-ranked journals operating across 22 scientific disciplines. We observed considerable heterogeneity across disciplines, with for example, 122/165 (74%) of assessed Health & Life Sciences journals providing statistical guidance and 0/30 (0%) of assessed Formal Sciences journals providing statistical guidance. We found that p-values and confidence intervals tended to be mentioned more often that Bayesian statistics; though when guidance on Bayesian statistics was provided, it generally indicated implicit endorsement. This study provides a stratum of empirical evidence upon which to base conceptual discussions about the role journals should play in the governance of statistical analysis.
Dissemination: Hardwicke et al. (preprint, https://doi.org/hmq6(odnośnik otworzy się w nowym oknie)). Hardwicke et al. (2022, presentation at the International Congress on Peer Review and Scientific Publication)
(3) We performed an extensive cross-disciplinary review of literature addressing the topic of preregistration. Our report outlines how preregistration can serve to reduce bias, increase transparency, and calibrating confidence in scientific claims.
Dissemination: Hardwicke & Wagenmakers (preprint, https://doi.org/f76t)(odnośnik otworzy się w nowym oknie); Hardwicke (2022 presentation at the Joint Statistical Meetings); Hardwicke (2021, presentation at Ludwig Maximilian University of Munich).
(4) We tested the feasibility of an intervention intended to detect undisclosed discrepancies during peer review and flag them to authors and editors so they could be resolved prior to publication. We found that the intervention could be successfully deployed and identified relevant parameters that could aid the preparation of a future randomized controlled trial to evaluate intervention efficacy.
Dissemination: TARG Meta-Research Group & Collaborators (2022, https://doi.org/h78p(odnośnik otworzy się w nowym oknie))
(5) We summarised the expansive literature on replication/reproducibility in psychology and provided new empirical evidence on several key issues.
Dissemination: Nosek, et al. (2022, https://doi.org/g3qf(odnośnik otworzy się w nowym oknie))
(6) Scientific studies often contain critical flaws, so it is important that the research community can criticise published research. In this study we assessed how 15 top-ranked journals in each of 22 scientific disciplines (330 journals in total) handled post-publication critique (for example 'letters to the editor). We found that just over a third (123) of journals did not accept post-publication critique, and those that did often imposed strict limits on length and time-to-submit. For example, one journal required that critiques were less than 175 words long and another journal required that critiques were submitted within two-weeks of the original study being published. We also found that journals rarely published critiques — based on an assessment of 2066 randomly sampled research articles, we estimated the prevalence of post-publication critique to be 1.9%, 95% confidence interval [1.4 2.6]. We also found that although most critiques attract a response from the original authors, those authors very rarely change their minds about the original conclusions. Overall, the study demonstrated that top-ranked journals are a serious barrier to the publication of critical discourse across scientific disciplines.
Dissemination: Hardwicke et al. (2022, https://doi.org/h828)(odnośnik otworzy się w nowym oknie); Hardwicke et al. (2022, presentation at International Congress on Peer Review and Scientific Publication); Hardwicke (2022, presentation at METRICS International Forum, Stanford University); Hardwicke (2022, presentation at Society for Improving Psychological Science Conference)
(7) We examined the trajectory of citation patterns and citation valence before and after four impactful psychology studies could not be replicated in large-scale preregistered replication projects. We observed only a slight post-replication decline in favourable citations and a small increase in unfavourable citations. This suggests that in practice, the expected ‘self-correction’ effect of replication studies can fail to materialize.
Dissemination: Hardwicke et al. (2021, https://doi.org/gwt8)(odnośnik otworzy się w nowym oknie); Hardwicke (2022, presentation at Berkeley Initiative for Transparency in the Social Sciences Annual Meeting)
(8) We observed that of 140 papers retracted in psychology due to data fabrication, scientific misconduct, or error, after retraction, 88 (63%) of these papers received at least one positive citation (median = 2, interquartile range = 4, min = 0, max = 89 positive citations). This suggests that interventions are needed to improve researchers’ awareness of retractions (e.g. notifying them through their paper management software).
Dissemination: Fernández et al. (preprint, https://doi.org/hcht(odnośnik otworzy się w nowym oknie))
The quality of scientific research has serious societal implications: poor quality research wastes resources and can lead to ineffective or even harmful policy interventions or clinical treatments. The research performed during this project has expanded our understanding of these important issues beyond the state of the art, for example by highlighting how 'self-correction' mechanisms like replication and retraction are often ineffective, how many (reconsolidation) studies are poorly designed and analyzed, and how journals often neglect transparency, statistical guidance, and post-publication critique.