Optimal control aims to develop decision making algorithms to extract the maximal benefit out of a dynamical system. Optimal control problems arise in a range of application domains, including energy management (where the aim is to meet energy demand with the minimum cost/carbon footprint under constraints imposed by the dynamics of the underlying physical processes), or portfolio optimisation (where the aim could be to maximise return subject to the dynamics and uncertainties of the markets), to name but a few. In the absence of accurate models for the underlying processes, optimal control problems are sometimes treated in a data-driven fashion. This could be in the spirit of reinforcement learning (where optimal decisions are derived by observing the effect of earlier actions and the resulting rewards) or Model Predictive Control (where optimal decisions are derived by using historical system data in lieu of a model). Despite wide-ranging progress on both the theory and applications of optimal control for more than half a century, considerable challenges remain when it comes to applying the resulting methods to large-scale systems. The difficulties become even greater when one moves outside the classical realm of model-based optimal control to address problems where models are replaced by data, or macroscopic behaviours emerge out of microscopic interactions of large populations of agents.
OCAL addressed precisely this challenge, by developing a framework for approximately solving optimal control problems that is both computationally tractable and provides theoretical approximation guarantees. In the context of approximate dynamic programming, the starting point were formulations of optimal control problems linear programs. Since for continuous states and action these programs are infinite dimensional, we developed randomised methods relying on finite dimensional function approximation and the sampling of constraints as a basis for algorithms. Our approach enjoys close connections to statistical learning theory, providing a direct link to data-driven approximation and resulting in the desired theoretical guarantees. Besides uncovering theoretical properties of these methods, however, our work showed that scaling them up to large-scale systems is far from trivial computationally, as empirically it requires an unreasonably large number of constraints to ensure that the approximate linear program remains bounded. We addressed this issue by moving away from random constraint sampling and developing structured, iterative constraint sampling methods. This nicely complemented our parallel on approximate solution of dynamic programming problems for finite state-action problems, where high performance, parallel software was developed for performing the approximation, drawing on a theoretical connection to non-smooth variants of Newton’s method. To demonstrate the efficacy of these methods, in addition to benchmark problems we also applied them to a simulation case study on insulin injection for the treatment of diabetes.
In the context of model predictive control, our work focused on the use of data to alleviate the need to develop a model. We showed that, though primarily inspired by deterministic linear problems where it is exact, this approach can also be used to approximate nonlinear and stochastic problems through regularisation. We were moreover able to establish a close link between the choice of regulariser and the various sources of uncertainty entering the problem. The approximation methodology resulted in a very powerful method that we were able to apply to practical problems, both in simulation and in experiments; examples range from quadrotor control in the lab, to energy management and urban traffic management.
