Project description
Mapping the hidden vulnerabilities of AI
While AI is changing our world, weaknesses in algorithms can create risks. These include adversarial attacks and biased outcomes. The ERC-funded NumAStAI project aims to find, measure, and reduce these vulnerabilities through numerical analysis. The research focuses on understanding likely attack scenarios, enabling subtle edits and fixes, assessing risks to underrepresented data categories, exposing low-cost adversarial strategies, analysing errors from low-precision calculations, and producing clear results to guide AI legislation. By combining matrix analysis, optimisation, and probabilistic methods, the project seeks to uncover AI’s hidden weaknesses and improve ethical, secure, and reliable deployment. Ultimately, NumAStAI will help ensure AI is both effective and trustworthy.
Objective
From a numerical analysis perspective I will identify, quantify and mitigate vulnerabilities in current artificial intelligence (AI) algorithms.
Novel mathematical research will emerge along six overlapping axes:
Inevitability: rigorously understand the inescapable endgame of the attack-versus-defence paradigm. Under what conditions is it inevitable that adversaries will
succeed? Formalizing such conditions will allow us to understand and, where possible, overcome current AI instabilities.
Editability: study algorithms that stealthily change a small number of parameters. This scenario is highly pertitent when new AI is built on top of third-party, foundation models. It also opens up the
possibility of fixing errors on-the-fly without the need to re-train.
Targetability: examine whether under-represented categories in the training data are more susceptible to adversarial attacks. This topic raises a key,
and currently overlooked, issue in the ethical use of AI.
Universality: develop linear, sparse, low rank mappings that
create adversarial attacks. These new functions will expose novel, low-cost, threat
mechanisms, but will also give insights into the decision boundary landscape.
Roundability: use state-of-the art probabilistic rounding error analysis to
justify large-scale, low precision computations.
I will study (a) why current AI technologies appear to defy traditional worst-case
floating point error bounds, and (b) whether low precision can be exploited by
an adversary.
Legislatability: devise easy-to-interpret results and practical case studies that can inform public opinion and impact the design of appropriate AI legislation.
The project identifies high-profile open questions requiring tools from
matrix analysis, optimization, backward error analysis, condition number theory and
stochastic computation. Some of the proposed work is highly speculative and challenging,
but will significantly advance our understanding of AI vulnerabilities.
Fields of science (EuroSciVoc)
CORDIS classifies projects with EuroSciVoc, a multilingual taxonomy of fields of science, through a semi-automatic process based on NLP techniques. See: The European Science Vocabulary.
CORDIS classifies projects with EuroSciVoc, a multilingual taxonomy of fields of science, through a semi-automatic process based on NLP techniques. See: The European Science Vocabulary.
This project has not yet been classified with EuroSciVoc.
Be the first one to suggest relevant scientific fields and help us improve our classification service
You need to log in or register to use this function
Keywords
Project’s keywords as indicated by the project coordinator. Not to be confused with the EuroSciVoc taxonomy (Fields of science)
Project’s keywords as indicated by the project coordinator. Not to be confused with the EuroSciVoc taxonomy (Fields of science)
Programme(s)
Multi-annual funding programmes that define the EU’s priorities for research and innovation.
Multi-annual funding programmes that define the EU’s priorities for research and innovation.
-
HORIZON.1.1 - European Research Council (ERC)
MAIN PROGRAMME
See all projects funded under this programme
Topic(s)
Calls for proposals are divided into topics. A topic defines a specific subject or area for which applicants can submit proposals. The description of a topic comprises its specific scope and the expected impact of the funded project.
Calls for proposals are divided into topics. A topic defines a specific subject or area for which applicants can submit proposals. The description of a topic comprises its specific scope and the expected impact of the funded project.
Funding Scheme
Funding scheme (or “Type of Action”) inside a programme with common features. It specifies: the scope of what is funded; the reimbursement rate; specific evaluation criteria to qualify for funding; and the use of simplified forms of costs like lump sums.
Funding scheme (or “Type of Action”) inside a programme with common features. It specifies: the scope of what is funded; the reimbursement rate; specific evaluation criteria to qualify for funding; and the use of simplified forms of costs like lump sums.
HORIZON-ERC - HORIZON ERC Grants
See all projects funded under this funding scheme
Call for proposal
Procedure for inviting applicants to submit project proposals, with the aim of receiving EU funding.
Procedure for inviting applicants to submit project proposals, with the aim of receiving EU funding.
(opens in new window) ERC-2024-ADG
See all projects funded under this callHost institution
Net EU financial contribution. The sum of money that the participant receives, deducted by the EU contribution to its linked third party. It considers the distribution of the EU financial contribution between direct beneficiaries of the project and other types of participants, like third-party participants.
EH8 9YL Edinburgh
United Kingdom
The total costs incurred by this organisation to participate in the project, including direct and indirect costs. This amount is a subset of the overall project budget.