Skip to main content

From FUnction-based TO MOdel-based automated probabilistic reasoning for DEep Learning

Periodic Reporting for period 1 - FUN2MODEL (From FUnction-based TO MOdel-based automated probabilistic reasoning for DEep Learning)

Reporting period: 2019-10-01 to 2021-03-31

Machine learning – the science of building systems from data – is revolutionising computer science and artificial intelligence (AI). Much of its success is due to deep neural networks, which have demonstrated outstanding performance in perception tasks such as image classification. Solutions based on deep learning are now being deployed in a multitude of real-world systems, from virtual personal assistants, through automated decision making in business, to self-driving cars.

The performance of deep learning systems has been shown to match human perception ability in ‘narrow’ artificial intelligence tasks, but falls short in comparison with ‘strong AI’ that aims to match the levels of human intelligence. Machine learning is an essential technology to enable artificial agents, but it typically only learns associations expressed as non-linear functions and lacks the ability to reason causally about interventions, counterfactuals and ‘what if’ scenarios. Progress towards true artificial intelligence requires the consideration of cognitive models of decision making that incorporate inference from data, but go significantly beyond, in that they account for agents’ cognitive state, beliefs, goals and intentions, as well as agent interactions, in presence of uncertainty and partial observability.

Deep neural networks have good generalisation properties, but are unstable with respect to so called adversarial examples, where a small modification to input causes a misclassification, often with high confidence. Adversarial examples are still poorly understood but attracting attention because of the potential risks to safety and security in applications, as demonstrated in the attached image, where modifications to an image of a traffic sign results in a dangerous misclassification. Additionally, well publicised failures of deep learning systems, such as Uber’s fatal accident, Amazon’s automated hiring system and Microsoft’s bot, have raised concerns about their safety, fairness and transparency.

To enable ‘strong AI’ we need modelling frameworks for decision making for autonomous agents that encompass both statistical machine learning as well as logic-based reasoning. For such ‘strong AI’ to be safe, robust and fair, rigorous modelling and verification techniques are necessary to provide provable guarantees on the correct behaviour of the agents.

There are two major challenges that currently stand in the way. The first is conceptual, and centred on how the interaction between human and robotic agents should be devised to result in enhanced, mutually beneficial collaboration between humans and artificial agents, and how perception and cognitive aspects such as preferences and goals that inevitably influence human decisions are accounted for in these interactions. The second is primarily technical: how to devise a comprehensive, unified modelling framework that supports scalable, compositional reasoning, where both machine learning and logic-based reasoning are first class citizens, and how the key correctness properties can be expressed and verified.

This project will develop a model-based, probabilistic reasoning framework for autonomous agents with cognitive aspects, which supports reasoning about their decisions, agent interactions and inferences that capture cognitive information, in presence of uncertainty and partial observability. The FUN2MODEL project objectives are:

1. Develop automated probabilistic verification and synthesis techniques to guarantee safety, robustness and fairness for complex decisions based on machine learning.

2. Formulate a game-based modelling framework for studying systems of autonomous agents with cognitive aspects and their coordination.

3. Develop a probabilistic compositional framework for quantitative reasoning about the behaviour of systems of autonomous agents with cognitive aspects.

4. Implement and evaluate the techniques on a variety of case studies with respect to safety, trust, accountability and fairness.

The outcome would be a comprehensive set of theories, algorithms and software tools for modelling, verification and synthesis of collaborating, human and artificial autonomous agents. The software developed as part of the project will be open source, built as an extension of PRISM ( where practically feasible, and the modelling language, online tutorial, case studies, demonstrations, publications and lectures will be made available for download.

If successful, the project will result in major advances in the quest towards provably robust and beneficial AI.
The FUN2MODEL project has made a slower than expected start due the unprecedented challenges of Covid-19. The project has nevertheless made steady progress, aided by key outcomes delivered just before the project formally started, listed under Key Prior Publications on the project website in addition to Publications developed during this period. In the initial phase the main emphasis has been on automated verification for deep learning, including techniques for safety and robustness evaluation for more general classes of neural networks (LSTMs), robust explainability, and applications in computer vision and NLP. A novel probabilistic formulation of safety verification was also proposed for Bayesian neural networks and developed through a series of publications, together with applications in control and progress towards correct-by-construction synthesis. Partial progress has been made in data-driven modelling, including adversarial robustness for Gaussian process models, partial observability in PRISM models, incorporated within PRISM software release 4.7 and causality, including a novel formulation of robustness to causal interventions for Bayesian networks. Successful outcomes have been achieved regarding modelling, verification and strategy synthesis to support coordination of multi-agent systems, and specifically equilibria verification for concurrent stochastic games, with applications in security and communication protocols. The techniques have been implemented and disseminated through a software release of PRISM-games 3.0.

The highlights include papers at leading conferences in verification (CAV), machine learning (ICML, AISTATS), AI (AAAI, UAI, IJCAI), as well as computer vision and NLP (CVPR, EMNLP). The PI has given several (mostly virtual) keynotes at leading conferences, including CONCUR, Ubicomp, ASE, ERTS, VMCAI, DNA and KR. Parker gave a keynote at iFM.

The PI has been invited to the Global Partnership on Artificial Intelligence (GPAI) Working Group on Responsible AI, nominated by the European Commission, was a panelist at the Royal Society's Briefing for Making Europe a Leader in AI, and was a member of the Royal Society’s Working Group on Digital Technology and the Planet.

Since the proposal has been submitted, the PI has been recognised through election as a Fellow of the Royal Society, ELLIS Fellow, the Lovelace Award, and the title of Professor awarded by the President in her home country of Poland.
The FUN2MODEL project has developed strong foundations on which to further advance the science towards provably beneficial collaborations between human and artificial agents. In future, emphasis wiill be given to cognitive modelling, neurosymbolic agents, game-theoretic models that can capture mental attitudes, causality, fairness and applications.
Adversarial examples for traffic signs taken from