A Theory for Understanding, Designing, and Training Deep Learning Systems

Información del proyecto

THUNDEEP

Identificador del acuerdo de subvención: 754705

DOI

10.3030/754705

Proyecto cerrado

Fecha de la firma de la CE 5 Septiembre 2017

Fecha de inicio 1 Septiembre 2018

Fecha de finalización 31 Julio 2025

Financiado con arreglo a

• EXCELLENT SCIENCE - European Research Council (ERC)

Coste total

€ 1 442 360,00

Aportación de la UE

€ 1 442 360,00

Coordinado por

WEIZMANN INSTITUTE OF SCIENCE
Israel

Periodic Reporting for period 5 - THUNDEEP (A Theory for Understanding, Designing, and Training Deep Learning Systems)

Período documentado: 2024-09-01 hasta 2025-07-31

Resumen del contexto y de los objetivos generales del proyecto

Deep learning, in the form of artificial neural networks, is one of the most rapidly evolving fields in machine learning, with wide-ranging impact on real-world applications. Neural networks can efficiently represent complex predictors, and are nowadays routinely trained successfully. Unfortunately, our scientific understanding of neural networks is quite rudimentary. Most methods used to design and train these systems are based on rules-of-thumb and heuristics, and there is a drastic theory-practice gap in our understanding of why these systems actually work. We believe this poses a significant risk to the long-term health of the field, as well as an obstacle to widening the applicability of deep learning beyond that achieved with current methods. The goal of this project is to develop principled tools for understanding, designing, and training deep learning systems, based on rigorous theoretical results. This is a major challenge in this rapidly evolving field, and any progress along these lines is expected to have a substantial impact on the theory and practice of creating such systems. To do so, we focus on three inter-related sources of performance losses in neural network learning: The optimization error of neural networks (that is, how to train a given network in a computationally efficient manner); The estimation error (how to ensure that training a network on a finite training set will ensure good performance on future examples); and the approximation error (how architectural choices of the networks affect the type of functions they can compute).

Trabajo realizado desde el comienzo del proyecto hasta el final del período abarcado por el informe y los principales resultados hasta la fecha

The project made a substantial impact on the research community’s understanding of deep learning and artificial neural networks, and led to significant advances across all of the projects objectives. Notable highlights include a much better understanding of the role of the network’s depth and width on the type of functions that they can express; a more rigorous understanding of deep learning system’s ability to successfully generalize (and the types of generalization behaviors that can be expected), as well as new practical algorithms pertaining to deep learning, such as data set reonstruction from trained neural networks. In addition, the project has led to several successful offshoots in the context of non-convex and convex optimization, which underlies the training of neural networks, such as the ability to find stationary points and the performance of stochastic gradient-based methods across several settings.

Avances que van más allá del estado de la técnica e impacto potencial esperado (incluida la repercusión socioeconómica y las implicaciones sociales más amplias del proyecto hasta la fecha)

By the end of the project, we have significantly advanced our rigorous understanding of deep learning, and placed it on a much firmer theoretical footing. The different objectives we have pursued resulted in new insights about how to design and train deep learning systems in practice.

reconstruction.png