Fueled by an exponential growth in data and compute power, in recent years, we have witnessed breathtaking advances in the field of machine learning (ML). This has led to a technology race in industry, with ML models being adopted in an ever increasing array of applications. As a consequence, there is a lot of enthusiasm in deploying ML models in increasingly high-stakes applications, such as self-driving cars, medical applications etc., where ML algorithms are used to autonomously make decisions in the real world. This results in a departure from the most well studied realm of ML, namely supervised learning (where the goal is to learn to predict), and enters the domain of reinforcement learning (RL, where the goal is to learn to act). This abstract model considers an agent who seeks to make decisions in an uncertain world. Since the agent does not know how the world works, it faces the dilemma of trading exploration – conducting experiments to better understand the consequences of its actions and associated rewards – and exploitation – using what it learned to make effective decisions. Similar to supervised learning RL has experienced dramatic breakthroughs in recent years, including DeepMind’s AlphaGo’s & AlphaZero’s landmark victories in the game of go. Strikingly, most of these successes are in games: Perfectly controlled environments, where – given enough computational power – virtually unlimited exploration is possible. In most real-world systems, characterized by high complexity and large amounts of uncertainty, however, at best approximate simulators are available. As a consequence, learning has to happen, at least partially, based on observations from real, physical systems. Suddenly the notion of exploration becomes a dangerous proposition: It means experimenting with actions whose consequences are uncertain. This fact disqualifies most existing approaches for RL, which utilize unconstrained – and possibly unsafe – exploration.
The RADDICS project developed novel RL algorithms that are provably reliable, even when deployed on high-stakes applications. Our approach hinges upon marrying nonparametric statistical learning with robust optimization. In particular, we use Bayesian approaches to quantify uncertainty in the prediction, in a way that yields valid high-probability confidence estimates about the unknown dynamics and rewards, even under some possibly adversarial circumstances. We then act safely under all plausible models, by employing tools from robust optimization and control theory. Additional observations contract the posterior, allowing to learn and improve policies over time in a safe manner. Beyond developing new algorithms and theory, we demonstrated our approach on several real-world applications, ranging from robotics to scientific applications such as safely tuning the SwissFEL Free Electron Laser, a large scientific facility operated by the Paul Scherrer Institute.
