Verifiably Safe and Correct Deep Neural Networks

Informazioni relative al progetto

VeriDeL

ID dell’accordo di sovvenzione: 101112713

DOI

10.3030/101112713

Data della firma CE 25 Ottobre 2023

Data di avvio 1 Novembre 2023

Data di completamento 31 Ottobre 2028

Finanziato da

• European Research Council (ERC)

Costo totale

€ 1 500 000,00

Contributo UE

€ 1 500 000,00

Coordinato da

THE HEBREW UNIVERSITY OF JERUSALEM
Israel

Periodic Reporting for period 1 - VeriDeL (Verifiably Safe and Correct Deep Neural Networks)

Periodo di rendicontazione: 2023-11-01 al 2026-04-30

Sintesi del contesto e degli obiettivi generali del progetto

Artificial intelligence (AI), and especially deep neural networks, now plays an increasingly important role in many areas of society. These systems can analyze data, recognize patterns, and make decisions with a level of accuracy that often surpasses traditional approaches. They are already used in applications such as autonomous vehicles, medical imaging, robotics, finance, and digital services. However, as AI becomes more deeply embedded in systems that affect people’s safety and wellbeing, a major challenge has emerged: many of these models behave as “black boxes”. It is extremely difficult to understand why they make certain decisions, and even more difficult to guarantee that they will act safely and reliably in all situations.

This lack of transparency limits the use of AI in the domains where it could bring the greatest benefit. For example, a navigation system for a vehicle or a robot must offer strong guarantees that it will not make catastrophic mistakes. Healthcare decision-support tools must behave consistently and fairly even in unusual circumstances. To deploy AI responsibly, society needs methods to verify its correctness, explain its behavior, and manage risks when the model is uncertain or vulnerable.

The project was created to address these needs by developing new scientific foundations and practical tools for verifying the behavior of deep neural networks and for ensuring their trustworthy use in safety-critical settings. Its objectives are threefold:
(1) to design faster and more reliable verification algorithms that can analyze modern, large-scale neural networks;
(2) to create abstraction techniques that simplify complex models in a mathematically sound way, allowing verification to scale; and
(3) to apply these techniques in real-world scenarios such as robotics, aerospace systems, and learning-based control, demonstrating how AI systems can be made safer, more transparent, and easier to certify.

By combining insights from computer science, mathematics, and AI safety, the project aims to provide tools that enable engineers, regulators, and developers to understand and trust the behavior of machine-learning models. Its expected impact includes improving the safety of AI-enabled technologies, supporting future certification processes, and contributing to Europe’s broader strategy for human-centered and trustworthy AI.

Lavoro eseguito dall’inizio del progetto fino alla fine del periodo coperto dalla relazione e principali risultati finora ottenuti

During the reporting period, the project advanced all major scientific and technical objectives.

A first line of work focused on improving verification algorithms for deep neural networks. The project developed more efficient procedures for analyzing how networks behave under all relevant inputs. These include enhanced search strategies, improved mathematical optimization routines, and dedicated techniques to handle the non-linear components that make neural networks challenging to verify. The project also introduced mechanisms for producing machine-checkable proofs of correctness, allowing verification results to be independently validated.

A second line of work produced new abstraction–refinement techniques. These methods build simplified versions of a network that retain essential behavior while being far easier to analyze. If a property holds for the abstract model, it is guaranteed to hold for the full system. When the abstraction is too coarse, the procedure automatically refines it. This approach makes it possible to reason about networks that were previously too large to verify. The project extended this idea to the verification of explanations, enabling the identification of input features that fully determine a decision, together with mathematical guarantees of their correctness.

A third line of work applied verification to deep reinforcement learning (DRL) and other real-world scenarios. The project developed tools to measure how often DRL agents are vulnerable to small perturbations and how these weaknesses evolve over time. It also introduced a lightweight, verification-guided “shield” that protects an agent during operation, activating only when necessary and allowing safe behavior without significant performance loss.
In the aerospace domain, the project created a statistical method for checking whether the outputs of AI-based classifiers are reliable, enabling early detection of potentially unsafe predictions.

Across these activities, the project delivered new theoretical insights - such as complexity analyses identifying which types of explanations or guarantees are feasible - as well as practical software tools and case studies. Together, these advances demonstrate the feasibility of trustworthy AI systems supported by rigorous verification.

Progressi oltre lo stato dell’arte e potenziale impatto previsto (incluso l’impatto socioeconomico e le implicazioni sociali più ampie del progetto fino ad ora)

The project has produced results that advance current scientific and technological capabilities in several ways.

First, the verification algorithms developed within the project considerably expand the size and complexity of neural networks that can be analysed with mathematical guarantees. This represents a significant step beyond existing tools, which struggle with large or highly non-linear models. The introduction of independently checkable proof objects further enhances trust in verification outcomes, addressing a key barrier to adoption in regulated industries.

Second, the project’s abstraction–refinement methods open new possibilities for verifying networks that were previously out of reach. Extending these techniques to explanation-related properties goes beyond current interpretability methods by providing provable and minimal explanations rather than heuristic ones. This creates opportunities for safer and more transparent AI systems in areas such as healthcare, decision-support, and automated control.

Third, the project contributes new tools for analysing and improving the safety of learning-based control systems. The Adversarial Rate metric, verification-guided shielding, and aerospace-focused certification techniques all represent innovative and practical approaches that can be integrated into future AI development pipelines. These results are particularly relevant for autonomous robotics, aviation, and other applications where reliability is essential.

Looking ahead, further uptake of these results will benefit from continued research, engineering refinement, and - in some sectors - supportive regulatory frameworks that recognize formal verification as part of the certification process. There are also opportunities for collaboration with industry to integrate verification tools into existing development workflows, and for international standardization bodies to incorporate formal guarantees into emerging AI safety standards.

Overall, the project strengthens Europe’s ability to build trustworthy AI systems and provides foundations that can support safe and responsible innovation in the years ahead.