Periodic Reporting for period 1 - Alexander Krauss (The limits of the sciences in identifying causes and scientific laws)
Reporting period: 2017-03-15 to 2019-03-14
Scientists do not often question the scientific methods used to generate causes in their particular scientific models. This research project investigates leading scientific methods, especially statistical methods, and their implicit theory, assumptions and limitations. It then outlines ways to reduce them. Better understanding the limits of using such scientific methods is important for research and policy, because these methods can often lead to some degree of biased results and scientists using them at times misguidedly claim to establish causal relationships. Yet they at times still inform policymakers and practitioners and can thus bring about negative medical, technological and societal outcomes. By combining theoretical, methodological and empirical analysis, this research project aims to help disentangle the links between the actual methods applied by scientists and the causal effects they claim to identify using these methods. The conclusions include that researchers and policymakers need to become better aware of the issues facing studies they use to inform policy, and that we need to better combine different research methods to improve understanding.
The output of this research project focused on three interrelated objectives and produced three respective research articles. These are outlined in the following:
Objective 1 (and corresponding article 1 that has already been published): Randomised controlled trials (RCTs) are commonly used across the sciences and viewed as the best research method to inform public health and social policy. Usually they are thought of as providing the most rigorous evidence of a treatment’s effectiveness without strong assumptions, biases and limitations. The objective is to examine that hypothesis by assessing the ten most cited RCT studies worldwide. These ten RCT studies with the highest number of citations in any journal were identified by searching Scopus (the largest database of peer-reviewed journals). In terms of the results, this study shows that these world-leading RCTs that have influenced policy produce biased results by illustrating that participants’ background traits that affect outcomes are often poorly distributed between trial groups, that the trials often neglect alternative factors contributing to their main reported outcome and, among many other issues, that the trials are often only partially blinded or unblinded. The study also identifies a number of novel and important assumptions, biases and limitations not yet thoroughly discussed in existing studies that arise when designing, implementing and analysing trials. Overall, trials involve complex processes – from randomising and controlling, to monitoring participants etc. – that require many decisions and steps at different levels that bring their own assumptions and degree of bias to results. In terms of conclusions, researchers and policymakers need to become better aware of the broader set of assumptions, biases and limitations in trials. Journals need to also begin requiring researchers to outline them in their studies.
Objective 2 (and corresponding article 2 that is under review): In the medical, behavioural and social sciences, the leading method used to model causal relationships is randomised controlled trials that are generally thought of as both the source and justification of the most valid evidence. In studying the foundation and theory behind RCTs, some leading scholars oversimplify the ability of the RCT method to deduce valid causal conclusions. They often discuss one individual issue at a time (such as randomisation, statistical probabilities, placebos and the like). To address the central question about the overall validity of RCTs we must however assess the broader range of important methodological issues, together, that affect the estimated causal results in trials when carried out in practice. I thereby provide a broader account of biased causal inference and overall validity in RCTs that explains how trials are not able to produce precise causal results. The implications include that, for many questions, we must not overly rely on any single scientific method but use multiple methods.
Objective 3 (and corresponding article 3 that is under review): Medical students and practitioners often lack sufficient training and understanding in the range of methodological biases and constraints facing randomised experiments. They are often constrained in assessing the outcomes and conclusions reported in medical studies, which they nonetheless at times use to inform their everyday decisions. Yet poorly informed decisions can lead to negative medical outcomes. I outline the range of complex factors that arise in conducting medical studies, including in selecting a sample, generating particular variables, randomising and blinding, implementing interventions and ensuring adherence, gathering data and dealing with unknown factors, interpreting conclusions, among many others. In this context, some of the most commonly discussed reasons for problems in reproducing medical studies are poor sample quality, selective reporting and pressure to publish. Such factors apply to some studies but not others, while there is one factor not commonly discussed that generally applies to all studies across the medical sciences: I thereby argue that the reproducibility crisis is, to a large extent, the result of the scientific process always being a highly complex human process involving many such steps – with some degree of bias unavoidably arising during this process. Variation between study outcomes is thus the norm, and 1 to 1 replication across study contexts is not possible. A central implication is that medical educators need to improve the methodological training given to the next generation of medical practitioners so they are better informed about the foundations and limitations of study results and the associated problems in reproducing study results – and about how to interpret and use study results in everyday medical practice. If practitioners do not however fully understand the range of methodological issues and problems facing studies then they will have to just rely on the information and warning labels provided by biopharmaceutical companies selling the tested treatments.
Objective 1 (and corresponding article 1 that has already been published): Randomised controlled trials (RCTs) are commonly used across the sciences and viewed as the best research method to inform public health and social policy. Usually they are thought of as providing the most rigorous evidence of a treatment’s effectiveness without strong assumptions, biases and limitations. The objective is to examine that hypothesis by assessing the ten most cited RCT studies worldwide. These ten RCT studies with the highest number of citations in any journal were identified by searching Scopus (the largest database of peer-reviewed journals). In terms of the results, this study shows that these world-leading RCTs that have influenced policy produce biased results by illustrating that participants’ background traits that affect outcomes are often poorly distributed between trial groups, that the trials often neglect alternative factors contributing to their main reported outcome and, among many other issues, that the trials are often only partially blinded or unblinded. The study also identifies a number of novel and important assumptions, biases and limitations not yet thoroughly discussed in existing studies that arise when designing, implementing and analysing trials. Overall, trials involve complex processes – from randomising and controlling, to monitoring participants etc. – that require many decisions and steps at different levels that bring their own assumptions and degree of bias to results. In terms of conclusions, researchers and policymakers need to become better aware of the broader set of assumptions, biases and limitations in trials. Journals need to also begin requiring researchers to outline them in their studies.
Objective 2 (and corresponding article 2 that is under review): In the medical, behavioural and social sciences, the leading method used to model causal relationships is randomised controlled trials that are generally thought of as both the source and justification of the most valid evidence. In studying the foundation and theory behind RCTs, some leading scholars oversimplify the ability of the RCT method to deduce valid causal conclusions. They often discuss one individual issue at a time (such as randomisation, statistical probabilities, placebos and the like). To address the central question about the overall validity of RCTs we must however assess the broader range of important methodological issues, together, that affect the estimated causal results in trials when carried out in practice. I thereby provide a broader account of biased causal inference and overall validity in RCTs that explains how trials are not able to produce precise causal results. The implications include that, for many questions, we must not overly rely on any single scientific method but use multiple methods.
Objective 3 (and corresponding article 3 that is under review): Medical students and practitioners often lack sufficient training and understanding in the range of methodological biases and constraints facing randomised experiments. They are often constrained in assessing the outcomes and conclusions reported in medical studies, which they nonetheless at times use to inform their everyday decisions. Yet poorly informed decisions can lead to negative medical outcomes. I outline the range of complex factors that arise in conducting medical studies, including in selecting a sample, generating particular variables, randomising and blinding, implementing interventions and ensuring adherence, gathering data and dealing with unknown factors, interpreting conclusions, among many others. In this context, some of the most commonly discussed reasons for problems in reproducing medical studies are poor sample quality, selective reporting and pressure to publish. Such factors apply to some studies but not others, while there is one factor not commonly discussed that generally applies to all studies across the medical sciences: I thereby argue that the reproducibility crisis is, to a large extent, the result of the scientific process always being a highly complex human process involving many such steps – with some degree of bias unavoidably arising during this process. Variation between study outcomes is thus the norm, and 1 to 1 replication across study contexts is not possible. A central implication is that medical educators need to improve the methodological training given to the next generation of medical practitioners so they are better informed about the foundations and limitations of study results and the associated problems in reproducing study results – and about how to interpret and use study results in everyday medical practice. If practitioners do not however fully understand the range of methodological issues and problems facing studies then they will have to just rely on the information and warning labels provided by biopharmaceutical companies selling the tested treatments.
The results and impacts of the project are outlined in the previous section. In terms of the dissemination activities, I have presented the results of this research at a number of international academic conferences – including in Belgium, Spain and Germany.
In terms of public engagement activities, I published a blog for the LSE Impact Blog ‘Contrary to common belief, randomised controlled trials inevitably produce biased results’, and for The Campbell Collaboration ‘Trials and errors: The limits of randomised controlled trials’
In terms of news coverage for the published article (article 1) ‘Why all randomised controlled trials produce biased results’:
- A briefing was published in Nature (Nature Briefing) ‘Randomized controlled trials have bias, too’
- An article was published in the British Medical Journal (BMJ) ‘Randomised controlled trials may have many unrecognised potential biases’
- A report featuring this study was published in the German science magazine, Bild der Wissenschaft ‘Der Videobeweis der Medizin’
The study is also used as a public resource for researchers: ‘INSEAD Randomized Controlled Trials Lab’
In terms of public engagement activities, I published a blog for the LSE Impact Blog ‘Contrary to common belief, randomised controlled trials inevitably produce biased results’, and for The Campbell Collaboration ‘Trials and errors: The limits of randomised controlled trials’
In terms of news coverage for the published article (article 1) ‘Why all randomised controlled trials produce biased results’:
- A briefing was published in Nature (Nature Briefing) ‘Randomized controlled trials have bias, too’
- An article was published in the British Medical Journal (BMJ) ‘Randomised controlled trials may have many unrecognised potential biases’
- A report featuring this study was published in the German science magazine, Bild der Wissenschaft ‘Der Videobeweis der Medizin’
The study is also used as a public resource for researchers: ‘INSEAD Randomized Controlled Trials Lab’