A first manuscript entitled "Bayesian Bilinear Neural Network for Predicting the Mid-price Dynamics in Limit-Order Book Markets" addresses the first major research question.
The paper explores the applicability of a Bayesian version of a Neural network parametrized by a Temporal Augmented Bilinear Layer (TABL), for forecasting mid-price movements in limit order books.
We adopt the state-of-the-art VOGN optimizer for the Bayesian training of the TABL parameters and the standard non-Bayesian ADAM optimizer. We show that the Bayesian training of the 464 parameters characterizing our model is indeed feasible and satisfactory. We then turn to evaluating and comparing the performance of the two approaches showing that, though the results are similar, the Bayesian TABL perhaps generalized better on unseen test data: we discuss how to construct forecasts from the predictive distribution, how to interpret the probabilities associated with the foretasted price movements and argue that from a practical perspective these probabilities (measures of uncertainties accosted with the model's point-predictions) are useful in making investment-related decisions.
In "Quasi Black-Box Variational Inference with Natural Gradients for Bayesian Learning" we develop an algorithm that merges elements of the BBVI framework and NGVI update. Indeed, the majority of VI algorithms do require the computation of models' gradients, which is, in the general, an expensive task. On the other hand, Black-box methods are capable of performing optimization without requiring the model's gradients, but via function queries only. The advantage is that the BB framework is readily applicable without requiring model-specific gradients or derivations. On the other hand, the NGVI setup devises the use of natural gradients for performing SGD-like updates within a theoretical foundation that enormously simplifies their computation. Natural gradients have been extensively studied and shown to be preferable to common Euclidean gradients in Bayesian optimization. On this basis, we develop an optimization routine capable of performing VI in a black-box fashion by relying on natural gradient computations. We test out optimizer on a number of datasets and statistical-econometric models showing alignment with maximum likelihood results, excellent accordance between the variational posteriors and the true one approximate via Monte Carlo Markov Chain sampling, and a variety of existing methods available in the literature.
Besides QBVI we developed an alternative black-box procedure based on manifold optimization. We develop on the so-called Manifold Gaussian Variational Bayes (MGVB) method (Tran, 2019) capable of explicitly addressing the symmetric and positive-definite constraint on the covariance matrix of the Gaussian variational distribution. Our optimizer uses exact natural gradient computations. Indeed, whereas MGVB uses some approximate results, the same theoretical results used in QBVI turn out to be adaptable to this context allowing for exact natural gradient computations. Opposed to QBVI and earlier methods, our EMGVB approach, because of the constrained manifold optimization, guarantees positive-semidefinite updates of the variational covariance matrix resulting in a robust and efficient procedure, less susceptible to the specification o the hyper-parameters. We validate our approach along with several alternative ones over different economic datasets and perform extensive analyses over different variants of the GARCH volatility model.
In a final manuscript, we review the state-of-the-art Bayesian machine learning techniques. For a reader not acquainted with the literature in the area, we present a multitude of approaches and algorithms that are feasible for Bayesian inference in machine learning applications. Besides describing standard methods such as Monte-Carlo samplers and Monte-Carlo dropout we focus on algorithms for Variational Inference, including those developed within this action. As of now, an algorithm-oriented literature review on Bayesian methods for ML is not available: our contribution is expected to promote such methods to a wider audience and provide a step-by-step introduction to the different approaches developed in the last decade.
