(1) A systematic review of 168 human reconsolidation experiments found that few studies meeting the design conditions necessary to provide a robust test of reconsolidation theory. Additionally, a subset of studies that have employed the desired 2x2 factorial design, often fall foul to a statistical fallacy in which a statistically significant different between two groups, and the absence of a statistically significant difference between two other groups, is taken as an indication that there is a significant interaction effect. Bayesian reanalyses that actually test the interaction in these contexts found that there was insufficient evidence to conclude an interaction effect, thus challenging the conclusions of the original papers.
(2) We sought to describe the nature and prevalence of statistical guidance provided to authors at 330 top-ranked journals operating across 22 scientific disciplines. We observed considerable heterogeneity across disciplines, with for example, 122/165 (74%) of assessed Health & Life Sciences journals providing statistical guidance and 0/30 (0%) of assessed Formal Sciences journals providing statistical guidance. We found that p-values and confidence intervals tended to be mentioned more often that Bayesian statistics; though when guidance on Bayesian statistics was provided, it generally indicated implicit endorsement. This study provides a stratum of empirical evidence upon which to base conceptual discussions about the role journals should play in the governance of statistical analysis.
Dissemination: Hardwicke et al. (preprint,
https://doi.org/hmq6(opens in new window)). Hardwicke et al. (2022, presentation at the International Congress on Peer Review and Scientific Publication)
(3) We performed an extensive cross-disciplinary review of literature addressing the topic of preregistration. Our report outlines how preregistration can serve to reduce bias, increase transparency, and calibrating confidence in scientific claims.
Dissemination: Hardwicke & Wagenmakers (preprint,
https://doi.org/f76t)(opens in new window); Hardwicke (2022 presentation at the Joint Statistical Meetings); Hardwicke (2021, presentation at Ludwig Maximilian University of Munich).
(4) We tested the feasibility of an intervention intended to detect undisclosed discrepancies during peer review and flag them to authors and editors so they could be resolved prior to publication. We found that the intervention could be successfully deployed and identified relevant parameters that could aid the preparation of a future randomized controlled trial to evaluate intervention efficacy.
Dissemination: TARG Meta-Research Group & Collaborators (2022,
https://doi.org/h78p(opens in new window))
(5) We summarised the expansive literature on replication/reproducibility in psychology and provided new empirical evidence on several key issues.
Dissemination: Nosek, et al. (2022,
https://doi.org/g3qf(opens in new window))
(6) Scientific studies often contain critical flaws, so it is important that the research community can criticise published research. In this study we assessed how 15 top-ranked journals in each of 22 scientific disciplines (330 journals in total) handled post-publication critique (for example 'letters to the editor). We found that just over a third (123) of journals did not accept post-publication critique, and those that did often imposed strict limits on length and time-to-submit. For example, one journal required that critiques were less than 175 words long and another journal required that critiques were submitted within two-weeks of the original study being published. We also found that journals rarely published critiques — based on an assessment of 2066 randomly sampled research articles, we estimated the prevalence of post-publication critique to be 1.9%, 95% confidence interval [1.4 2.6]. We also found that although most critiques attract a response from the original authors, those authors very rarely change their minds about the original conclusions. Overall, the study demonstrated that top-ranked journals are a serious barrier to the publication of critical discourse across scientific disciplines.
Dissemination: Hardwicke et al. (2022,
https://doi.org/h828)(opens in new window); Hardwicke et al. (2022, presentation at International Congress on Peer Review and Scientific Publication); Hardwicke (2022, presentation at METRICS International Forum, Stanford University); Hardwicke (2022, presentation at Society for Improving Psychological Science Conference)
(7) We examined the trajectory of citation patterns and citation valence before and after four impactful psychology studies could not be replicated in large-scale preregistered replication projects. We observed only a slight post-replication decline in favourable citations and a small increase in unfavourable citations. This suggests that in practice, the expected ‘self-correction’ effect of replication studies can fail to materialize.
Dissemination: Hardwicke et al. (2021,
https://doi.org/gwt8)(opens in new window); Hardwicke (2022, presentation at Berkeley Initiative for Transparency in the Social Sciences Annual Meeting)
(8) We observed that of 140 papers retracted in psychology due to data fabrication, scientific misconduct, or error, after retraction, 88 (63%) of these papers received at least one positive citation (median = 2, interquartile range = 4, min = 0, max = 89 positive citations). This suggests that interventions are needed to improve researchers’ awareness of retractions (e.g. notifying them through their paper management software).
Dissemination: Fernández et al. (preprint,
https://doi.org/hcht(opens in new window))
