The output of this research project focused on three interrelated objectives and produced three respective research articles. These are outlined in the following:
Objective 1 (and corresponding article 1 that has already been published): Randomised controlled trials (RCTs) are commonly used across the sciences and viewed as the best research method to inform public health and social policy. Usually they are thought of as providing the most rigorous evidence of a treatment’s effectiveness without strong assumptions, biases and limitations. The objective is to examine that hypothesis by assessing the ten most cited RCT studies worldwide. These ten RCT studies with the highest number of citations in any journal were identified by searching Scopus (the largest database of peer-reviewed journals). In terms of the results, this study shows that these world-leading RCTs that have influenced policy produce biased results by illustrating that participants’ background traits that affect outcomes are often poorly distributed between trial groups, that the trials often neglect alternative factors contributing to their main reported outcome and, among many other issues, that the trials are often only partially blinded or unblinded. The study also identifies a number of novel and important assumptions, biases and limitations not yet thoroughly discussed in existing studies that arise when designing, implementing and analysing trials. Overall, trials involve complex processes – from randomising and controlling, to monitoring participants etc. – that require many decisions and steps at different levels that bring their own assumptions and degree of bias to results. In terms of conclusions, researchers and policymakers need to become better aware of the broader set of assumptions, biases and limitations in trials. Journals need to also begin requiring researchers to outline them in their studies.
Objective 2 (and corresponding article 2 that is under review): In the medical, behavioural and social sciences, the leading method used to model causal relationships is randomised controlled trials that are generally thought of as both the source and justification of the most valid evidence. In studying the foundation and theory behind RCTs, some leading scholars oversimplify the ability of the RCT method to deduce valid causal conclusions. They often discuss one individual issue at a time (such as randomisation, statistical probabilities, placebos and the like). To address the central question about the overall validity of RCTs we must however assess the broader range of important methodological issues, together, that affect the estimated causal results in trials when carried out in practice. I thereby provide a broader account of biased causal inference and overall validity in RCTs that explains how trials are not able to produce precise causal results. The implications include that, for many questions, we must not overly rely on any single scientific method but use multiple methods.
Objective 3 (and corresponding article 3 that is under review): Medical students and practitioners often lack sufficient training and understanding in the range of methodological biases and constraints facing randomised experiments. They are often constrained in assessing the outcomes and conclusions reported in medical studies, which they nonetheless at times use to inform their everyday decisions. Yet poorly informed decisions can lead to negative medical outcomes. I outline the range of complex factors that arise in conducting medical studies, including in selecting a sample, generating particular variables, randomising and blinding, implementing interventions and ensuring adherence, gathering data and dealing with unknown factors, interpreting conclusions, among many others. In this context, some of the most commonly discussed reasons for problems in reproducing medical studies are poor sample quality, selective reporting and pressure to publish. Such factors apply to some studies but not others, while there is one factor not commonly discussed that generally applies to all studies across the medical sciences: I thereby argue that the reproducibility crisis is, to a large extent, the result of the scientific process always being a highly complex human process involving many such steps – with some degree of bias unavoidably arising during this process. Variation between study outcomes is thus the norm, and 1 to 1 replication across study contexts is not possible. A central implication is that medical educators need to improve the methodological training given to the next generation of medical practitioners so they are better informed about the foundations and limitations of study results and the associated problems in reproducing study results – and about how to interpret and use study results in everyday medical practice. If practitioners do not however fully understand the range of methodological issues and problems facing studies then they will have to just rely on the information and warning labels provided by biopharmaceutical companies selling the tested treatments.
