Periodic Reporting for period 2 - Neuroimaging power (Effect size and power for neuroimaging.)
Reporting period: 2018-11-01 to 2019-10-31
The action "Neuroimaging power" looks at the problem of statistical power for neuroimaging studies, with concrete proposals for improvements.
There is increasing concern about reproducibility in published neuroscience research. A major contributor to this problem is the prevalence of low statistical power. That is, findings from a low-power study are less likely to be reproducible and thus a power analysis is a critical component of any paper. Power calculations prevent researchers from spending time and money on studies that are underpowered, but on the other hand also prevent wasting time and money adding extra subjects, when sufficient power was already available. However, accurate power analyses for neuroimaging studies are rare, and general knowledge about effect sizes are scarce.
Objectives of this Marie Sklodowska Curie Action has been to (1) increase knowledge about accurate effect sizes, (2) provide intuition about effect sizes for different tasks and brain regions and (3) predict statistical power for neuroimaging data.
During this project, we expanded the availability and useability of tools to estimate and increase statistical power for neuroimaging studies.
There is increasing concern about reproducibility in published neuroscience research. A major contributor to this problem is the prevalence of low statistical power. That is, findings from a low-power study are less likely to be reproducible and thus a power analysis is a critical component of any paper. Power calculations prevent researchers from spending time and money on studies that are underpowered, but on the other hand also prevent wasting time and money adding extra subjects, when sufficient power was already available. However, accurate power analyses for neuroimaging studies are rare, and general knowledge about effect sizes are scarce.
Objectives of this Marie Sklodowska Curie Action has been to (1) increase knowledge about accurate effect sizes, (2) provide intuition about effect sizes for different tasks and brain regions and (3) predict statistical power for neuroimaging data.
During this project, we expanded the availability and useability of tools to estimate and increase statistical power for neuroimaging studies.
Work was conducted via 3 aims, translated into work packages.
Our first aim was to increase knowledge about accurate effect sizes. This has been operationalised with the development of a toolbox to estimate effect sizes from prior studies and incorporate this metric into a sample size estimator for statistical designs. We have extended our power analysis toolbox neuropower to include standardised effect sizes, estimated based on the distribution of peak heights observed in the statistical map of a neuroimaging experiment. The toolbox allows neuroscientists to upload their neuroimaging statistical maps and conduct an effect size estimation and subsequent power analysis.
We have noted that power analyses on statistical maps are not generalisable to other experimental designs, and therefore inherently incomparable. To overcome this incomparability issue, we have implemented a method to estimate design efficiency from an experimental design. We have written this efficiency calculator in a python library and a web application neurodesign. These computational tools not only estimate the efficiency, but also help researchers optimising their experimental designs.
The first work package resulted in 1 manuscript accepted for publications, 2 invited presentations and 1 conference presentation.
The second aim was to provide intuition about effect sizes for different tasks and brain regions. To this end, we have integrated our power analysis toolbox with neurovault, a repository for statistical images. This allows researchers to directly estimate effect sizes and power for any publicly available image in the repository.
Furthermore, we have estimated effect sizes for data from the human connectome project and for the consortium for neuropsychiatric phenomics, two very rich datasets in terms of variability. We have written a pipeline to estimate effect sizes in specific regions based on the raw data. We have applied this pipeline to a published dataset from the Human Connectome Project, which was used to provide insight in the scale of effect sizes observed in neuroimaging. We have shown a current overview of statistical power in neuroimaging studies, by aggregating sample sizes from over 1000 studies over more than 20 years, see Figure 1 (Poldrack et al. 2017, Nature).
Work package 2 resulted in 2 journal publications and the publication of open source data pipelines.
Lastly, we have continued ongoing work on predicting statistical power for neuroimaging data. We have further improved our power analysis estimation method, to exclude the need to set an exclusion threshold.
Work package 3 comprised of 1 journal publication and 3 invited conference presentations.
Our first aim was to increase knowledge about accurate effect sizes. This has been operationalised with the development of a toolbox to estimate effect sizes from prior studies and incorporate this metric into a sample size estimator for statistical designs. We have extended our power analysis toolbox neuropower to include standardised effect sizes, estimated based on the distribution of peak heights observed in the statistical map of a neuroimaging experiment. The toolbox allows neuroscientists to upload their neuroimaging statistical maps and conduct an effect size estimation and subsequent power analysis.
We have noted that power analyses on statistical maps are not generalisable to other experimental designs, and therefore inherently incomparable. To overcome this incomparability issue, we have implemented a method to estimate design efficiency from an experimental design. We have written this efficiency calculator in a python library and a web application neurodesign. These computational tools not only estimate the efficiency, but also help researchers optimising their experimental designs.
The first work package resulted in 1 manuscript accepted for publications, 2 invited presentations and 1 conference presentation.
The second aim was to provide intuition about effect sizes for different tasks and brain regions. To this end, we have integrated our power analysis toolbox with neurovault, a repository for statistical images. This allows researchers to directly estimate effect sizes and power for any publicly available image in the repository.
Furthermore, we have estimated effect sizes for data from the human connectome project and for the consortium for neuropsychiatric phenomics, two very rich datasets in terms of variability. We have written a pipeline to estimate effect sizes in specific regions based on the raw data. We have applied this pipeline to a published dataset from the Human Connectome Project, which was used to provide insight in the scale of effect sizes observed in neuroimaging. We have shown a current overview of statistical power in neuroimaging studies, by aggregating sample sizes from over 1000 studies over more than 20 years, see Figure 1 (Poldrack et al. 2017, Nature).
Work package 2 resulted in 2 journal publications and the publication of open source data pipelines.
Lastly, we have continued ongoing work on predicting statistical power for neuroimaging data. We have further improved our power analysis estimation method, to exclude the need to set an exclusion threshold.
Work package 3 comprised of 1 journal publication and 3 invited conference presentations.
We concluded this project with a collection of new tools and methods to estimate experimental designs and power for neuroimaging studies. This MSCA has pushed the frontiers of neuroimaging science, by providing new tools to evaluate studies. The effect size estimator can be reused in new technologies for power analysis or study reporting, while the software integration allows for easy inspection of existing studies. The power analysis tool provides researchers with an easy tool, implementing an advanced method to evaluate their neuroimaging studies, before, during and after the study.
With this, the neuroimaging community has new tools to improve the current reproducibility crisis.
With this, the neuroimaging community has new tools to improve the current reproducibility crisis.