Periodic Reporting for period 4 - GTBB (General theory for Big Bayes)
Berichtszeitraum: 2023-09-01 bis 2025-03-31
In the modern era of complex and large data sets, there is stringent need for flexible, sound and scalable inferential methods to analyze them. Bayesian approaches have been increasingly used in statistics and machine learning and in all sorts of applications such as biostatistics, astrophysics, social science etc. Major advantages of Bayesian approaches are: their ability to model complex models in a hierarchical way, their coherency and ability to deliver not only point estimators but also measures of uncertainty from the posterior distribution which is a probability distribution on the parameter space at the core of all Bayesian inference. The increasing complexity of the data sets raise huge challenges for Bayesian approaches: theoretical and computational and we are now at a turning point where the tool which have made Bayesian approaches so successful in the last 20 years have reached their limits and new directions need to be considered. The aim of this project is to develop a general theory for the analysis of Bayesian methods in complex and high (or infinite) dimensional models which will cover not only fine understanding of the posterior distributions but also an analysis of the output of the algorithms used to implement the approaches.
The main objectives of the project are (briefly):
1. Asymptotic analysis of the posterior distribution of complex high dimensional models
2. Asymptotic analysis of approximate computational Bayes approaches – Interractions between asymptotics and computations.
The main objectives of the project are (briefly):
1. Asymptotic analysis of the posterior distribution of complex high dimensional models
2. Asymptotic analysis of approximate computational Bayes approaches – Interractions between asymptotics and computations.
We have developed models and theory for networks, deep neural networks and different stochastic processes. We have developped Bernstein von Mises properties for semiparametric models; we have studied and developped Bayesian methods for manifold learning These works have lead to 21 publications and 10 preprints and 7 PhD theses. All these results are available on arxiv.
1. Complex modelling and inference. We studied 4 families of complex models: Point processes, Networks, Diffusion models and Deep Neural networks. We also investigated Bayesian manifold learning.
Point processes: We developed theory and methodology for Bayesian nonparametric methods for multivariate nonlinear Hawkes processes and estimation of the graph of interactions: published in Bernoulli and under minor revision at JMLR. We developed and studied Bayesian nonparametric methods for spatial point processes with covariates, published in Bernoulli. We also used discrete times Hawkes processes to model the early stage of the Covid epidemic, published in PLoSone and for conflict data, submitted on arxiv.
Networks. We developed new generative models for networks, we studied the asymptotic behaviour of a wide class of models called the graphex models and the asymptotic behaviour of a Bayesian approach. These have been published in Bernoulli, Advances in applied probability, EJS and ECML PKDD.
diffusion models: We studied Bayesian nonparametric methods for estimating the drift vector field in a multi-dimensional, published in the Annals of Statistics.
Deep neural networks: We studied the impact of initialisation in the large width large depth regimes, leading to 3 publications in AISTAT, Neurips workshop, TMLR and 1 preprint. We also studied the interplay between penalties and priors in the context of DNN, submitted on arxiv. We compared Gaussian processes and deep learning approaches for Gaussian regression, showing the superiority of the latter in some situations, published in Neurips. We studied convergence rates of score based diffusion generative models based on DNN under the manifold hypothesis, submitted on arxiv..
Bayesian manifold learning. We developed and studied a Bayesian procedure for estimating the density living near a manifold. We also studied regression under the manifold hypothesis, under both extrinsic an intrinsic Gaussian process priors. We obtained minimax rates of convergence for Bayesian nonparametric regression using based graph Laplacians in large dimension,. These works have been published respectively in Annals of Statistics, Neurips and (minor revision) JMLR.
2.Semiparametric inference
(i) Non standard losses: We studied concentration of the posterior distribution under local losses; published in JASA and Bernoulli. We also studied the multivariate deconvolution problem in max-sliced Wasserstein, from a frequentist a Bayesian perspective, published in the Annals of Statistics.
(ii) Bernstein von Mises theorems (BvM): we proposed two different Bayesian quasi posterior methods to ensure that the resulting distributions behave well for multiple targets, published in EJS and in JRSS B. Finally we derived a BvM for a class on nonparametric mixture models, submitted on arxiv.
(iii) Empirical Bayes. We characterised the behaviour of the Maximum marginal likelihood estimator in various irregular scenarios
3. Approximate Bayesian approaches:
(i) We derived robust Bayesian methods using Generalized Bayesian inference, published in AISTAT. We also generalised the notion of minimax convergence rates under random sample sizes,submitted on arxiv.
(ii) We studied 3 families of approximate Bayesian computations: coresets, ABC and variational algorithms, published in Neurips, arxiv and JMLR (under minor revision) respectively.
(ii) We studied efficient algorithms to compute the marginal likelihoods in mixture models investigating also distributed inference, published in a thesis.
1. Complex modelling and inference. We studied 4 families of complex models: Point processes, Networks, Diffusion models and Deep Neural networks. We also investigated Bayesian manifold learning.
Point processes: We developed theory and methodology for Bayesian nonparametric methods for multivariate nonlinear Hawkes processes and estimation of the graph of interactions: published in Bernoulli and under minor revision at JMLR. We developed and studied Bayesian nonparametric methods for spatial point processes with covariates, published in Bernoulli. We also used discrete times Hawkes processes to model the early stage of the Covid epidemic, published in PLoSone and for conflict data, submitted on arxiv.
Networks. We developed new generative models for networks, we studied the asymptotic behaviour of a wide class of models called the graphex models and the asymptotic behaviour of a Bayesian approach. These have been published in Bernoulli, Advances in applied probability, EJS and ECML PKDD.
diffusion models: We studied Bayesian nonparametric methods for estimating the drift vector field in a multi-dimensional, published in the Annals of Statistics.
Deep neural networks: We studied the impact of initialisation in the large width large depth regimes, leading to 3 publications in AISTAT, Neurips workshop, TMLR and 1 preprint. We also studied the interplay between penalties and priors in the context of DNN, submitted on arxiv. We compared Gaussian processes and deep learning approaches for Gaussian regression, showing the superiority of the latter in some situations, published in Neurips. We studied convergence rates of score based diffusion generative models based on DNN under the manifold hypothesis, submitted on arxiv..
Bayesian manifold learning. We developed and studied a Bayesian procedure for estimating the density living near a manifold. We also studied regression under the manifold hypothesis, under both extrinsic an intrinsic Gaussian process priors. We obtained minimax rates of convergence for Bayesian nonparametric regression using based graph Laplacians in large dimension,. These works have been published respectively in Annals of Statistics, Neurips and (minor revision) JMLR.
2.Semiparametric inference
(i) Non standard losses: We studied concentration of the posterior distribution under local losses; published in JASA and Bernoulli. We also studied the multivariate deconvolution problem in max-sliced Wasserstein, from a frequentist a Bayesian perspective, published in the Annals of Statistics.
(ii) Bernstein von Mises theorems (BvM): we proposed two different Bayesian quasi posterior methods to ensure that the resulting distributions behave well for multiple targets, published in EJS and in JRSS B. Finally we derived a BvM for a class on nonparametric mixture models, submitted on arxiv.
(iii) Empirical Bayes. We characterised the behaviour of the Maximum marginal likelihood estimator in various irregular scenarios
3. Approximate Bayesian approaches:
(i) We derived robust Bayesian methods using Generalized Bayesian inference, published in AISTAT. We also generalised the notion of minimax convergence rates under random sample sizes,submitted on arxiv.
(ii) We studied 3 families of approximate Bayesian computations: coresets, ABC and variational algorithms, published in Neurips, arxiv and JMLR (under minor revision) respectively.
(ii) We studied efficient algorithms to compute the marginal likelihoods in mixture models investigating also distributed inference, published in a thesis.
Complex models: We developped new theory and methodology for inference in multivariate Hawkes processes, spatial point processes with covariates and multivariate reversible diffusions. Our advances were both on theory and methodology.
Bayesian Manifold learning. We developped a mathematical framework to describe anisotropic smoothness along a manifold and proved minimax rates for Bayesian nonparametric Gaussian mixtures. We also obtained adaptive minimax rates for graph Laplacian approaches, constructing a new approximation strategy for regression functions in this context.
Deep neural network and Bayesian inference: We showed for various architectures that initialisation can have a dramatic effect on the learning ability of the network and we proposed a rescaling which allows for better stabilisation. We showed the inability of Gaussian processes to learn compositional functions. Finally we proved that diffusion generative models did not suffer the curse of the ambiant dimensional under the manifold hypothesis.
semi-parametric inference. We have proposed two alternative methods to construct posterior distributions in semiparametric problems to obtain efficient estimation and correct uncertainty for many functionals of interest and models
We obtained an inversion inequality which allows to understand better Wasserstein neighbourhood of mixing densities in deconvolution problems.
We also developed a theory to study posterior distributions for functions whose regularity is spatially varying.
We obtained an asymptotic description of the posterior distribution for a class of sparse networks which allows for uncertainty quantification, covering also the misspecified case.
We developped variational methods for large non linear Hawkes processes. We improved on existing coresets approaches to approximate Bayesian methods for large data sets studied more complex scenarios for ABC algorithms.
Bayesian Manifold learning. We developped a mathematical framework to describe anisotropic smoothness along a manifold and proved minimax rates for Bayesian nonparametric Gaussian mixtures. We also obtained adaptive minimax rates for graph Laplacian approaches, constructing a new approximation strategy for regression functions in this context.
Deep neural network and Bayesian inference: We showed for various architectures that initialisation can have a dramatic effect on the learning ability of the network and we proposed a rescaling which allows for better stabilisation. We showed the inability of Gaussian processes to learn compositional functions. Finally we proved that diffusion generative models did not suffer the curse of the ambiant dimensional under the manifold hypothesis.
semi-parametric inference. We have proposed two alternative methods to construct posterior distributions in semiparametric problems to obtain efficient estimation and correct uncertainty for many functionals of interest and models
We obtained an inversion inequality which allows to understand better Wasserstein neighbourhood of mixing densities in deconvolution problems.
We also developed a theory to study posterior distributions for functions whose regularity is spatially varying.
We obtained an asymptotic description of the posterior distribution for a class of sparse networks which allows for uncertainty quantification, covering also the misspecified case.
We developped variational methods for large non linear Hawkes processes. We improved on existing coresets approaches to approximate Bayesian methods for large data sets studied more complex scenarios for ABC algorithms.