Project description
New theory describes deep neural network learning from high-order correlations
Like humans, deep neural networks (DNNs) can learn from complex and detailed interrelationships among multiple data points (so-called high-order correlations or HOCs) extracting data-specific features. However, existing theoretical frameworks do not capture this ability. The ERC-funded beyond2 project aims to develop a theory that explains how and what DNNs learn from HOCs. To overcome the unrealistic assumption of Gaussian inputs in current theories, the project will extend methods from statistical physics and high-dimensional statistics to manage non-Gaussian input distributions. The team will train DNNs by stochastic gradient descent to reveal how they learn from HOCs efficiently and investigate the relationship of so-called principal components of HOCs to fundamental data properties.
Objective
Deep neural networks (DNNs) have revolutionised how we learn from data. Rather than requiring careful engineering and domain knowledge to extract features from raw data, DNNs learn the relevant features for a task automatically from data. In particular, high-order correlations (HOCs) of the data are crucial for both the performance of DNNs and the type of features they learn. However, existing theoretical frameworks cannot capture the impact of HOCs – they either study “lazy” regimes where DNNs do not learn data-specific features, or they rely on the unrealistic assumption of Gaussian inputs devoid of non-trivial HOCs.
beyond2 will develop a theory for how and what neural networks learn from the high-order correlations of their data. We break the problem into two parts:
(i) *How?* We analyse the learning dynamics of neural networks trained by stochastic gradient descent to unveil the mechanism by which they learn from HOCs efficiently (in terms of the minimum amount of training data / learning time required to attain satisfactory predictive performance).
(ii) *What?* Our preliminary research suggests that neural filters are primarily determined by the “principal components” of HOCs. We investigate how these principal components relate to fundamental data properties, such as symmetries of the inputs.
We attack these problems by extending methods from statistical physics and high-dimensional statistics to handle non-Gaussian input distributions. Studying the interplay between data structure and learning dynamics will allow understanding how specific learning mechanisms, like attention or recursion, are able to unwrap HOCs.
By shifting the focus from unstructured to non-Gaussian data models, beyond2 will yield new insights into the inner workings of neural networks. These insights will bring theory closer to practice and might facilitate the safe deployment of neural networks in high-stakes applications.
Fields of science (EuroSciVoc)
CORDIS classifies projects with EuroSciVoc, a multilingual taxonomy of fields of science, through a semi-automatic process based on NLP techniques. See: The European Science Vocabulary.
CORDIS classifies projects with EuroSciVoc, a multilingual taxonomy of fields of science, through a semi-automatic process based on NLP techniques. See: The European Science Vocabulary.
- natural sciences computer and information sciences artificial intelligence computational intelligence
You need to log in or register to use this function
Keywords
Project’s keywords as indicated by the project coordinator. Not to be confused with the EuroSciVoc taxonomy (Fields of science)
Project’s keywords as indicated by the project coordinator. Not to be confused with the EuroSciVoc taxonomy (Fields of science)
Programme(s)
Multi-annual funding programmes that define the EU’s priorities for research and innovation.
Multi-annual funding programmes that define the EU’s priorities for research and innovation.
-
HORIZON.1.1 - European Research Council (ERC)
MAIN PROGRAMME
See all projects funded under this programme
Topic(s)
Calls for proposals are divided into topics. A topic defines a specific subject or area for which applicants can submit proposals. The description of a topic comprises its specific scope and the expected impact of the funded project.
Calls for proposals are divided into topics. A topic defines a specific subject or area for which applicants can submit proposals. The description of a topic comprises its specific scope and the expected impact of the funded project.
Funding Scheme
Funding scheme (or “Type of Action”) inside a programme with common features. It specifies: the scope of what is funded; the reimbursement rate; specific evaluation criteria to qualify for funding; and the use of simplified forms of costs like lump sums.
Funding scheme (or “Type of Action”) inside a programme with common features. It specifies: the scope of what is funded; the reimbursement rate; specific evaluation criteria to qualify for funding; and the use of simplified forms of costs like lump sums.
HORIZON-ERC - HORIZON ERC Grants
See all projects funded under this funding scheme
Call for proposal
Procedure for inviting applicants to submit project proposals, with the aim of receiving EU funding.
Procedure for inviting applicants to submit project proposals, with the aim of receiving EU funding.
(opens in new window) ERC-2024-STG
See all projects funded under this callHost institution
Net EU financial contribution. The sum of money that the participant receives, deducted by the EU contribution to its linked third party. It considers the distribution of the EU financial contribution between direct beneficiaries of the project and other types of participants, like third-party participants.
34136 Trieste
Italy
The total costs incurred by this organisation to participate in the project, including direct and indirect costs. This amount is a subset of the overall project budget.