Project description
Discovering how deep learning actually works
Deep learning (DL) simulates a neural network by using multiple layers to autonomously learn from data. Despite being effective in many applications, its theoretical foundation remains unknown. Enter the ERC-funded UnderstandingDL project, which will adopt a three-pronged approach. Firstly, it will develop new models learnable by stochastic gradient descent (SGD), an optimisation algorithm used to train neural networks. This may produce new DL algorithms which are theoretically understood. Secondly, it will investigate the sample complexity of different neural network types and node connection strengths. This should explain how DL can have fewer examples than parameters but still generalise successfully. Finally, it will investigate neural network functionality, including depth benefits and the effects of deliberately corrupted data (so-called adversarial examples).
Objective
While extremely successful, deep learning (DL) still lacks a solid theoretical foundation.
In the last 5 years the PI focused almost entirely on DL theory, yielding a strong publication record with 7 papers at NeurIPS (the leading ML conference), including 2 spotlights (top 3% of submitted papers) and one oral (top 1%), 2 papers at ICLR (the leading DL conference), and 1 paper at COLT (the leading ML theory conference). These results are amongst the first that break a 20 years hiatus in NN theory, thereby giving some hope for a solid deep learning theory. This includes 1) the first poly-time learnability result for non-trivial function class by SGD on NN, 2) the first such result with near optimal rate, 3) new methodology to bound the sample complexity of NN, that established the first sample complexity bound that is sublinear in the number of parameters, under norm constraints that are valid in practice, 4) an explanation to the phenomena of adversarial examples.
We plan to go far beyond these and other results, and to build a coherent theory for DL, addressing all three pillars of learning theory:
Optimization: We plan to investigate the success of SGD in finding a good model, arguably the greatest mystery of modern deep learning. Specifically our goal is to understand what models are learnable by SGD on neural networks. To this end, we plan to come up with a new class of models that can potentially lead to new deep learning algorithms, with a solid theory behind them.
Statistical Complexity: We plan to crack the second great mystery of modern deep learning, which is their ability to generalize with fewer examples than parameters. Our plan is to investigate the sample complexity of classes of neural networks that are defined by bounds on the weights’ magnitude.
Representation: We plan to investigate functions that can be realized by NN. This includes classical questions such as the benefits of depth, as well as more modern aspects such as adversarial examples.
Programme(s)
Multi-annual funding programmes that define the EU’s priorities for research and innovation.
Multi-annual funding programmes that define the EU’s priorities for research and innovation.
-
HORIZON.1.1 - European Research Council (ERC)
MAIN PROGRAMME
See all projects funded under this programme
Topic(s)
Calls for proposals are divided into topics. A topic defines a specific subject or area for which applicants can submit proposals. The description of a topic comprises its specific scope and the expected impact of the funded project.
Calls for proposals are divided into topics. A topic defines a specific subject or area for which applicants can submit proposals. The description of a topic comprises its specific scope and the expected impact of the funded project.
Funding Scheme
Funding scheme (or “Type of Action”) inside a programme with common features. It specifies: the scope of what is funded; the reimbursement rate; specific evaluation criteria to qualify for funding; and the use of simplified forms of costs like lump sums.
Funding scheme (or “Type of Action”) inside a programme with common features. It specifies: the scope of what is funded; the reimbursement rate; specific evaluation criteria to qualify for funding; and the use of simplified forms of costs like lump sums.
HORIZON-ERC - HORIZON ERC Grants
See all projects funded under this funding scheme
Call for proposal
Procedure for inviting applicants to submit project proposals, with the aim of receiving EU funding.
Procedure for inviting applicants to submit project proposals, with the aim of receiving EU funding.
(opens in new window) ERC-2021-STG
See all projects funded under this callHost institution
Net EU financial contribution. The sum of money that the participant receives, deducted by the EU contribution to its linked third party. It considers the distribution of the EU financial contribution between direct beneficiaries of the project and other types of participants, like third-party participants.
91904 JERUSALEM
Israel
The total costs incurred by this organisation to participate in the project, including direct and indirect costs. This amount is a subset of the overall project budget.