In this project, we made great strides in developing new robust reinforcement learning methods that cope with the presence of significant rare events that occur with low probability but nonetheless have a big effect on expected performance. In addition, we also developed efficient techniques for closely related challenges in multi-agent and meta reinforcement learning problems.
Progress was made on two fronts: developing frequentist approaches to these problems, i.e. those based on frequentist statistics, as outlined in WP1 and WP2; and developing Bayesian approaches to these problems, i.e. those based on Bayesian statistics, as outlined in WP3.
On the frequentist side, we developed a new method called off-environment reinforcement learning (OFFER) which takes a coevolutionary approach to policy search. A policy gradient method is used to optimise the policy while a second optimisation process in parallel optimises a proposal distribution from which trajectories are sampled, in order to ensure sufficient focus on significant rare events. This dual optimisation approach carries out the coevolutionary approach sketched in WP1.
Furthermore, we extended this work by devising a fundamental improvement to the policy gradient approach that underlies OFFER. The new approach, called expected policy gradients (EPG), marginalises out the stochastically selected action, and can be used to speed up both halves of OFFER’s dual optimisation. We have obtained strong theoretical results for EPG as well as substantial empirical results. The work on EPG also inspired a new approach we call Fourier policy gradients, which uses Fourier analysis to unify a family of policy gradient methods and construct new methods as well.
Building on this momentum, we have also developed multi-agent variants of these frequentist approaches, allowing teams of agents to efficiently optimise their joint control policies by exploiting these same principles. This is a natural extension since coevolution takes an inherently multi-agent perspective on optimisation. We also developed universal value exploration, which leverages key ideas from successor features in reinforcement learning to perform more efficient and scalable exploration in multi-agent learning. They also include tensorised actors, which exploit tools from tensor decomposition (high-dimensional variants of matrix factorisation techniques) to improve multi-agent reinforcement learning. Furthermore, we developed a technique for regularised softmax which addresses a systematic overestimation bias in multi-agent reinforcement learning methods. Finally, we developed factored value functions for multi-agent learning by proposing deep coordination graphs as well as factored centralised critics to learn faster value functions that generalise better.
On the Bayesian side, we developed a new method called alternating optimisation and quadrature (ALOQ), which uses Bayesian optimisation and Bayesian quadrature to tackle the same problem as OFFER. In each optimisation step, Bayesian optimisation is used to select a policy to evaluate, and Bayesian quadrature is used to sample the trajectory on which that policy will be evaluated. We obtained strong empirical results for ALOQ on a number of benchmark tasks, substantially outperforming existing methods. In addition, in collaboration with roboticists in France, we applied ALOQ to challenging simulated robot control tasks such as the control of a robot arm with multiple joints and the control of a walking hexapod robot. We also developed a successor method to ALOQ that we call contextual policy optimisation that automatically selects a distribution over environment variables that enables a policy gradient method to maximise its one-step performance. We also developed Bayesian Bellman operators, a new approach to Bayesian reinforcement learning that avoids the need to learn a model while fixing foundational problems with existing model-free Bayesian reinforcement learning methods.
