From FUnction-based TO MOdel-based automated probabilistic reasoning for DEep Learning

Project Information

FUN2MODEL

Grant agreement ID: 834115

DOI

10.3030/834115

Project closed

EC signature date 12 April 2019

Start date 1 October 2019

End date 30 September 2025

Funded under

• EXCELLENT SCIENCE - European Research Council (ERC)

Total cost

€ 2 417 891,50

EU contribution

€ 2 417 890,00

Coordinated by

THE CHANCELLOR, MASTERS AND SCHOLARS OF THE UNIVERSITY OF OXFORD
United Kingdom

Periodic Reporting for period 3 - FUN2MODEL (From FUnction-based TO MOdel-based automated probabilistic reasoning for DEep Learning)

Reporting period: 2022-10-01 to 2024-03-31

Summary of the context and overall objectives of the project

Machine learning – the science of building systems from data – is revolutionising computer science and artificial intelligence (AI). Much of its success is due to deep neural networks, which have demonstrated outstanding performance in perception tasks such as image classification. Solutions based on deep learning are now being deployed in a multitude of real-world systems, from virtual personal assistants, through automated decision making in business, to self-driving cars.

While deep learning systems have been shown to match human perception ability in ‘narrow’ artificial intelligence tasks, they fall short in comparison with ‘strong AI’ that aims to match the levels of human intelligence. Machine learning is an essential technology to enable artificial agents, but typically only learns associations expressed as non-linear functions and lacks the ability to reason causally about interventions, counterfactuals and ‘what if’ scenarios. Progress towards true artificial intelligence requires the consideration of cognitive models of decision making that incorporate inference from data, but go significantly beyond, in that they account for agents’ cognitive state, beliefs, goals and intentions, as well as agent interactions, uncertainty and partial observability.

Deep neural networks have good generalisation properties, but are unstable with respect to so called adversarial examples, where a small, intentional input perturbation causes a false prediction, often with high confidence. Adversarial examples are still poorly understood but attracting attention because of the potential risks to safety and security in applications, as demonstrated in the attached image, where a minor adversarial modification to an image of a traffic sign results in a dangerous misclassification. Additionally, well publicised failures of deep learning systems, such as Uber’s fatal accident, Amazon’s automated hiring system and Microsoft’s bot, have raised concerns about their safety, fairness and transparency.

To enable ‘strong AI’ we need modelling frameworks for decision making for autonomous agents that encompass both statistical machine learning as well as logic-based reasoning. For such ‘strong AI’ to be safe, robust and fair, rigorous modelling and verification techniques are necessary to provide provable guarantees on the correct behaviour of the agents.

There are two major challenges that currently stand in the way. The first is conceptual, and centred on how the interaction between human and robotic agents should be devised to result in enhanced, mutually beneficial collaboration between humans and artificial agents, and how perception and cognitive aspects such as preferences and goals that inevitably influence human decisions are accounted for in these interactions. The second is primarily technical: how to devise a comprehensive, unified modelling framework that supports scalable, compositional reasoning, where both machine learning and logic-based reasoning are first class citizens, and how the key correctness properties can be expressed and verified.

This project will develop a model-based, probabilistic reasoning framework for autonomous agents with cognitive aspects, which supports reasoning about their decisions, interactions and inferences that capture cognitive information, in the presence of uncertainty and partial observability. The FUN2MODEL project objectives are:

1. Develop automated probabilistic verification and synthesis techniques to guarantee safety, robustness and fairness for complex decisions based on machine learning.

2. Formulate a game-based modelling framework for studying systems of autonomous agents with cognitive aspects and their coordination.

3. Develop a probabilistic compositional framework for quantitative reasoning about the behaviour of systems of autonomous agents with cognitive aspects.

4. Implement and evaluate the techniques on a variety of case studies with respect to safety, trust, accountability and fairness.

The outcome would be a comprehensive set of theories, algorithms and software tools for modelling, verification and synthesis of collaborating, human and artificial autonomous agents. The software developed as part of the project will be open source, built as an extension of PRISM (www.prismmodelchecker.org) where practically feasible, and the modelling language, online tutorial, case studies, demonstrations, publications and lectures will be made available for download.

If successful, the project will result in major advances in the quest towards provably robust and beneficial AI.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

The FUN2MODEL project has made steady progress despite the Covid-19 pandemic striking just five months into the start of the project. The main advances to date have been grouped into the following research themes (see project website www.fun2model.org/researchthemes.php for more information):

• Safety, Robustness and Fairness Guarantees. Algorithms to compute adversarial robustness guarantees for neural networks, which draw on convex relaxation techniques, have been developed and applied to safety assurance, fairness and Natural Language Processing (NLP). This includes a novel anytime method to generate symbolic under-approximation of the pre-image of a polyhedron output set, which allows for quantitative verification. Safe execution of deep reinforcement learning systems has been enabled through formal (probabilistic) verification techniques.
• Robustness Guarantees for Bayesian Neural Networks. This strand of project developed methods for adversarial robustness guarantees for Bayesian neural networks, a variant of probabilistic machine learning that admits uncertainty. The properties cover probabilistic and decision safety, reach-avoid and certifiable robustness (lower/upper bounds on the probability).
• Efficient Robust Learning. We studied the feasibility and efficiency of robust learning against evasion attacks from the computational learning theory standpoint to address the provably correct synthesis goal for machine learning components. The work has focused on seeking results that ensure efficiency of robust learning in the Boolean hypercube by imposing appropriate distributional assumptions and bounding the adversary’s power.
• Tractable Causal Inference and Reasoning. Novel methods that exploit probabilistic circuits such as product-sum networks have been developed to provide tractable methods to compute provable guarantees on robustness of Bayesian networks to causal interventions, also in the presence of data uncertainty, and structure learning that supports causality queries and sample-based convergence guarantees for reinforcement learning.
• Multi-agent Coordination and Collaboration. A comprehensive set of techniques for verification and strategy synthesis for concurrent stochastic games has been developed and implemented in the PRISM-games model checker, covering finite- and infinite-horizon properties and a large class of probabilistic and reward objectives. It enables equilibria synthesis for Nash and correlated equilibria with respect to two optimality criteria, social welfare and social fairness. The modelling framework has been extended with neural perception mechanisms and partial observability, and offline and online algorithms for optimal strategy synthesis for discounted cumulative rewards devised and implemented.
• Human-like Decision Making. A cognitive stochastic game model has been formulated and implemented using probabilistic programming, which supports model construction, inference and cognitive reasoning. New insight was obtained into social trust through the classical trust game played by human participants against a custom bot whose decision making was driven by the cognitive game framework. A POMDP framework was extended to capture trust-aware motion planning for human-robot collaboration and planning for interactions between humans and robots that ensures that the temporal goals are reached despite the agents acting with a “trembling hand”.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The FUN2MODEL project has developed strong foundations on which to further advance the science towards provably beneficial collaborations between human and artificial agents. In future, in addition to progress on safety and robustness, emphasis will be given to neuro-symbolic techniques, modelling with perception, integration of cognitive aspects within game-theoretic models, and trust-aware modelling of human-robot collaborations.

Adversarial examples for traffic signs taken from http://www.fun2model.org/papers/hkww17.pdf