Skip to main content
Go to the home page of the European Commission (opens in new window)
English en
CORDIS - EU research results
CORDIS

Markov Chain Monte Carlo using couplings toward scalable statistical inference

Project description

MCMC efficiency and accuracy using modern computing hardware

Data science has long struggled with approximating probabilities, a challenge that affects fields such as Bayesian statistics, classical hypothesis testing, and models with intractable likelihoods. The Markov chain Monte Carlo (MCMC) method offers a powerful approach by using a Markov chain to estimate probabilities, but its efficiency is increasingly misaligned with modern computing hardware developments. The ERC-funded UMCMC project will advance the unbiased Markov chain Monte Carlo (UMCMC) framework to enhance MCMC coordination with contemporary computing architectures. By refining this approach, the project aims to improve accuracy, scalability, and computational efficiency, making probabilistic inference more effective across various scientific and statistical applications.

Objective

This project develops a statistical toolbox for the approximation of probability distributions that commonly arise in data analysis. The problem of approximating probabilities arise in many tasks of data science: in Bayesian statistics and its many variants, in classical hypothesis testing with p-values, in likelihood-based methods when the model involves latent variables, in models with intractable likelihoods, in the construction of knockoffs for principled variable selection, for example. State-of-the-art methods for such approximations include Markov chain Monte Carlo (MCMC), where a Markov chain is generated in such a way that it converges to the probability of interest as the length of the sequence goes to infinity. This stands at odds with modern developments in computing hardware, which provide an increasing number of parallel processors, but where each process has a stagnating clock speed. Methods that are amenable to parallel computing must emerge to help scientists in all fields to make the most of their data. The project builds upon a framework called Unbiased Markov chain Monte Carlo (UMCMC), in which accuracy improves arbitrarily with the number of parallel runs. Each run involves the generation of coupled Markov chains for a random time horizon.

Part 1 develops UMCMC to realize its potential as a comprehensive basis for probabilistic computation on modern hardware. The project includes theoretical analyses of cost and measures of efficiency, and methodological innovations towards adaptive, efficient, robust and convenient computation.

Part 2 contributes to the applicability of UMCMC, by conceptualizing the design of coupled Markov transitions, and considering a number of challenging settings: distributions supported on submanifolds and their application in economics, distributions on graphs and their applications in the fight against malaria, and Bayesian nonparametric models for cell type deconvolution from transcriptomics data.

Fields of science (EuroSciVoc)

Keywords

Programme(s)

Topic(s)

Funding Scheme

HORIZON-ERC - HORIZON ERC Grants

See all projects funded under this funding scheme

Call for proposal

(opens in new window) ERC-2024-COG

See all projects funded under this call

Host institution

ASSOCIATION GROUPE ESSEC
Net EU contribution
€ 2 000 000,00
Address
AVENUE BERNARD HIRSCH
95021 Cergy Pontoise Cedex
France
Activity type
Higher or Secondary Education Establishments
Links
Total cost
€ 2 000 000,00

Beneficiaries (1)