Project description
Deep kernel adaptation could explain deep learning performance
Scientists currently lack sufficient understanding of the boundaries on learning imposed by dataset size and network width. This is critical in the context of the classical limit of learning in statistical physics given the rapid expansion of real datasets. With the support of the Marie Skłodowska-Curie Actions programme, the SOTA project aims to show that a network’s output can be predicted by a similarity measure between data points (an effective kernel) exhibiting low-dimensional adaptation to the dataset. Unifying ‘lazy’ learning (computation delayed until prediction) with ‘rich’ learning (active weight updates to extract features), SOTA will work to develop a mechanistic explanation, also analysing corrections inconsequential for the classical limit but highly relevant for larger datasets.
Objective
The AI revolution is under way, yet we still lack a thermodynamic understanding of deep learning, that is explaining what a network can learn in the limit of large dataset and large network width.
The central proposal of SOTA is that, the output of a deep network on real data can be predicted by an effective kernel - a similarity measure between data points - that shows simple low-dimensional adaptation to the dataset. This reconnects two regimes often seen as completely different: “lazy” learning (where a fixed kernel predictor explains outputs) and “rich” learning (where network features are plastic and little understood).
Indeed, SOTA argues that rich learning in fully-connected networks effectively reduces predictor variance, while almost no adaptation of the mean predictor is predicted and observed. Convolutional networks instead show local adaptations of the kernel that also change the mean predictor.
This is where the number of data samples is proportional to width, the classical limit in statistical physics of learning. Due to the abundance (or augmentation) of data in recent deep learning practice, also the quadratic sample-width limit will need to be investigated.
Based on these consequential observations, a three-pronged investigation is proposed:
- Comparison of predictions to large-scale Monte-Carlo experiments on real data.
- Mechanistic explanation through a proposed mechanism of auto-ensembling.
- Analysis of corrections to the effective action that are sub-leading in the proportional limit, but become relevant when data is more abundant.
SOTA disrupts widely held assumptions about rich learning, and makes major progress on a long-standing challenge: Explaining the performance of deep, nonlinear networks in the feature learning regime, on real data, through a simple, effective theory.
Fields of science (EuroSciVoc)
CORDIS classifies projects with EuroSciVoc, a multilingual taxonomy of fields of science, through a semi-automatic process based on NLP techniques. See: The European Science Vocabulary.
This project's classification has been human-validated.
CORDIS classifies projects with EuroSciVoc, a multilingual taxonomy of fields of science, through a semi-automatic process based on NLP techniques. See: The European Science Vocabulary.
This project's classification has been human-validated.
Keywords
Project’s keywords as indicated by the project coordinator. Not to be confused with the EuroSciVoc taxonomy (Fields of science)
Project’s keywords as indicated by the project coordinator. Not to be confused with the EuroSciVoc taxonomy (Fields of science)
Programme(s)
Multi-annual funding programmes that define the EU’s priorities for research and innovation.
Multi-annual funding programmes that define the EU’s priorities for research and innovation.
-
HORIZON.1.2 - Marie Skłodowska-Curie Actions (MSCA)
MAIN PROGRAMME
See all projects funded under this programme
Topic(s)
Calls for proposals are divided into topics. A topic defines a specific subject or area for which applicants can submit proposals. The description of a topic comprises its specific scope and the expected impact of the funded project.
Calls for proposals are divided into topics. A topic defines a specific subject or area for which applicants can submit proposals. The description of a topic comprises its specific scope and the expected impact of the funded project.
Funding Scheme
Funding scheme (or “Type of Action”) inside a programme with common features. It specifies: the scope of what is funded; the reimbursement rate; specific evaluation criteria to qualify for funding; and the use of simplified forms of costs like lump sums.
Funding scheme (or “Type of Action”) inside a programme with common features. It specifies: the scope of what is funded; the reimbursement rate; specific evaluation criteria to qualify for funding; and the use of simplified forms of costs like lump sums.
HORIZON-TMA-MSCA-PF-EF - HORIZON TMA MSCA Postdoctoral Fellowships - European Fellowships
See all projects funded under this funding scheme
Call for proposal
Procedure for inviting applicants to submit project proposals, with the aim of receiving EU funding.
Procedure for inviting applicants to submit project proposals, with the aim of receiving EU funding.
(opens in new window) HORIZON-MSCA-2025-PF
See all projects funded under this callCoordinator
Net EU financial contribution. The sum of money that the participant receives, deducted by the EU contribution to its linked third party. It considers the distribution of the EU financial contribution between direct beneficiaries of the project and other types of participants, like third-party participants.
43121 PARMA
Italy
The total costs incurred by this organisation to participate in the project, including direct and indirect costs. This amount is a subset of the overall project budget.