Reducing required sample size of new studies by effectively using historical data: new methods for evaluating mechanisms, conditions, and alternative explanations

Project Information

Data Synthesis Power

Grant agreement ID: 792119

DOI

10.3030/792119

Project terminated on 31 July 2019

EC signature date 26 March 2018

Start date 1 September 2018

End date 31 August 2020

Funded under

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Total cost

€ 165 598,80

EU contribution

€ 165 598,80

Coordinated by

UNIVERSITEIT UTRECHT
Netherlands

Periodic Reporting for period 1 - Data Synthesis Power (Reducing required sample size of new studies by effectively using historical data: new methods for evaluating mechanisms, conditions, and alternative explanations)

Reporting period: 2018-09-01 to 2020-08-31

Summary of the context and overall objectives of the project

Scientists, organizations, and governments are moving in the direction of sharing their data. This will lead to an unprecedented amount of available data that researchers could use as prior information in a Bayesian analysis. While Bayesian methods are a promising method for using the growing amount of available information to increase statistical power in research studies, there is a great need for developing guidelines for calibrating existing data to create accurate informative prior distributions. Bayesian methods that incorporate accurate prior information in the statistical analysis reduce the required sample size of the new study without decreasing the chance of detecting a true effect. However, prior data may be from a different population, measured using different instruments, and/or collected using different experimental procedures than those in the current study. Using the results from a prior study that differs from the current study in any of these respects as prior information in a Bayesian analysis can lead to biased results.

Several promising calibration methods have been proposed for linear regression analysis. In this project I proposed to extend and test these methods to models with third variables that reveal the conditions under which an effect exists (moderators), the mechanism through which one variable affects another (mediators), and variables that might explain an observed effect (confounders).

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

Due to the project being terminated early, the researcher completed only a chapter on the nature of exchangeability in social sciences studies and stared writing SAS code for a simulation study to examine the potential increase in statistical power with the use of exchangeable and non-exchangeable prior information.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

This project has been terminated early and the expected results were not fully achieved: unfortunately, 11 months is too short a time to make progress beyond the state of the field. One conference presentation was given in 2019.

What has been achieved in the 11 months: the sub-project on mediation analysis was started, and SAS code was written to examine different methods for downweighing prior information for mediation analysis. Progress beyond the current state of the art would have been made had the results from the simulation study been done and translated into guidelines for applied researchers. But when the project was terminated the code was still running (a five-month process).

Results that were not achieved due to early termination: no statistical methods for downweighing prior information for models with moderators and confounders were examined in simulation studies, and thus no guidelines and software were developed for these two statistical models.

The expected impact of the achieved aims of the project is a clearer understanding of when prior studies can be used in the statistical analysis without calibration and when calibration is necessary. Furthermore, the SAS code for the simulation study to evaluate resulting statistical properties of using exchangeable and non-exchangeable prior information in mediation analysis is currently running and the researcher will write a paper based on it. The expected impact of the paper are guidelines for researchers in the social sciences on when it is advisable to use existing studies in a Bayesian analysis.