Periodic Reporting for period 2 - Stein-ML (Stein’s method and functional inequalities in machine learning)
Berichtszeitraum: 2022-12-01 bis 2023-11-30
The aim of this research project was to advance the development of quality measures for approximations in machine learning and statistics, using the rich theoretical machinery of mathematical analysis and probability. The project has been concluded with five papers, each targeting a specific approximation. Specifically, together with collaborators, we have achieved the following goals. We have constructed fully computable quality guarantees for the Laplace approximation of a Bayesian posterior, with respect to a variety of useful divergences. We have also constructed a new functional-data goodness-of-fit test for Gaussian Process targets and measures absolutely continuous with respect to Gaussians. Moreover, we have provided a novel targeted accuracy diagnostic for distributional approximations. Furthermore, in another paper, we have proved a functional version of the celebrated de Jong Theorem describing the asymptotic behaviour of U-statistics. Finally, we have proved new Berry-Esseen bounds for vector-valued statistics of binomial processes, with respect to the convex distance.
Secondly, we have derived quality guarantees for a common approximate inference method, called the Laplace approximation. The idea of this method is to replace an intractable posterior in Bayesian inference with an appropriate Gaussian distribution. Our guarantees are fully computable from the data. They also control the crucial quantities which are most commonly reported by users of Bayesian inference - posterior means, posterior variances and posterior credible sets. The sample-size and dimension dependence of our guarantees has been shown to be such that it cannot be improved in the generality of our assumptions.
Thirdly, we have completed a project in which we design a new diagnostic tool for the quality of Bayesian approximations, including variational inference in particular. Our diagnostic produces bounds on the error of posterior functionals of interest, including, for instance, component-wise means or variances. This is in contrast to other existing methods which characterize the quality of the whole variational distribution. Indeed, the quality of the whole variational distribution is typically poor in realistic applications, even if specific posterior functionals are accurate. Our new method fills this gap.
Moreover, we have completed two additional probability theory projects which advance the theory of Gaussian approximation - both in finite- and infinite-dimensional contexts. We have proved a novel functional approximation of rescaled degenerate U-statistics with Gaussian processes. We have also provided a new method for bounding the convex distance between Gaussian distributions and functionals of binomial processes.
The above mentioned results were disseminated via publications in international journals, such as Probability Theory and Related Fields, Annals of Applied Probability and Bernoulli, and in the Proceedings of the 26th International Conference on Artificial Intelligence and Statistics (AISTATS). Moreover, the results were disseminated through a number of presentations at international conferences, workshops and seminars. Our results are available open access, which makes them ready to be exploited by other researchers or industry practitioners.