Practically Relevant Theory of Deep Learning

Projektinformationen

TheoryDL

ID Finanzhilfevereinbarung: 676774

DOI

10.3030/676774

Projekt abgeschlossen

EK-Unterschriftsdatum 7 Dezember 2015

Startdatum 1 Februar 2016

Enddatum 31 Januar 2021

Finanziert unter

• EXCELLENT SCIENCE - European Research Council (ERC)

Gesamtkosten

€ 1 342 500,00

EU-Beitrag

€ 1 342 500,00

Koordiniert durch

THE HEBREW UNIVERSITY OF JERUSALEM
Israel

Periodic Reporting for period 4 - TheoryDL (Practically Relevant Theory of Deep Learning)

Berichtszeitraum: 2020-08-01 bis 2021-01-31

Zusammenfassung vom Kontext und den Gesamtzielen des Projekts

"Since the proposal of this project has been written, the impact of Deep Learning on our everyday life has been greatly increased.
Translating text, searching and organizing images based on textual content, chat-bots, self-driving cars, are all examples of technologies which heavily rely on Deep Learning.
To the general audience, new technologies tend to look like a magic.
The unique situation in deep learning is that this technology looks like a magic even to data scientists.
The goal of the TheoryDL project is to demystify deep learning, by providing a deep (pun intended) theoretical understanding of this technology, and in particular, understanding its potential but also its limitations.
The significance of this goal is two folded. First, I believe that it is dangerous to rely on technology which we do not understand. Second, a better theoretical understanding should enable to improve existing algorithms. Of particular interest is to be able to come up with faster algorithms, which are not of brute-force nature. Current algorithms contain a lot of brute-force components, and therefore the ""power"" of using deep learning is focused around few industrial companies that have the data and computing resources. A better theoretical understanding may lead to a democratization of this technology.
"

Arbeit, die ab Beginn des Projekts bis zum Ende des durch den Bericht erfassten Berichtszeitraums geleistet wurde, und die wichtigsten bis dahin erzielten Ergebnisse

We have tackled the problem from several angles and the findings are summarized in several publications (see the publications list).
At a high level, maybe the most important result is the coupling between deep learning and gradient-based algorithms, where we have shown that analyzing neural networks independently of the algorithm that is being used to train them is not the right approach.
On this theme, we have first performed a systematic study of failures of deep learning.
People tend to run and tell about success stories, but failures are even more interesting as laying down the boundaries of a technology enables to better understand why and when it works. We have identified cases in which gradient based training of deep learning fails miserably. Interestingly, the failures are neither due to overfitting/underfitting nor due to spurious local minima or a plethora of saddle points. They are rather due to more subtle issues such as insufficient information in the gradients or bad signal to noise ratios.
This direction led us to an important observation: that weight sharing is crucial for optimization of deep learning. We proved that without weight sharing, deep learning can only essentially learn low frequencies, but completely fails to learn mid and high frequencies. Weight sharing enables some sort of coarse-to-fine training.
From there, we were able to define generative hierarchical models for which provably efficient algorithms, that actually work in practice, exist.
We continued with a series of papers, where at the end we were able to establish the foundations of a general theory of deep learnability based on gradient-based algorithms using the language of statistical queries.

Fortschritte, die über den aktuellen Stand der Technik hinausgehen und voraussichtliche potenzielle Auswirkungen (einschließlich der bis dato erzielten sozioökonomischen Auswirkungen und weiter gefassten gesellschaftlichen Auswirkungen des Projekts)

"We have published several results that go beyond the state of the art in the understanding of deep learning. Most recently:
- We identified the connection between approximation, depth separation and learnability in neural networks (Malach, Yehudai, S., Shamir, 2021)
- We have proved the well known ""lottery ticket hypothesis"", showing the pruning is all what you need for building a deep network (Malach, Yehudai, S., Shamir, 2021).
- We have derived Computational Separation Between Convolutional and Fully-Connected Networks (Malach & S., 2020)
- We have derived a general novel theory of deep learning connecting both hardness of approximation and hardness of learning through a new concept of the ""Variance"" of hypothesis classes (Malach & S., 2020)"

fractals.jpg