Computers are now able to process language so efficiently that they can answer relatively complicated questions or generate poems. This progress is primarily due to the development of artificial “deep neural networks”. Nowadays, “deep learning” is revolutionizing our life, prompting an economic battle between technological giants and profoundly impacting society. As attractive and performant as it is, however, many agree that deep learning is largely an empirical field that lacks a theoretical understanding of its capacity and limitations. The algorithms used to "train" these neural networks explore a high-dimensional and non-convex energy landscape that eludes most of the present theoretical methodology in learning theory. The hope is that with a better theoretical understanding, we could build safer, more reliable and more performant systems.
In this project, we use advanced methods of statistical mechanics to develop a theoretical understanding of deep neural networks and their behaviour. We develop simplified models where learning performance can be analyzed and predicted mathematically. The overall goal is to make these models as realistic as possible and capture an extensive range of the behaviour observed empirically in deep learning. Analyzing how the performance depends on various tunable parameters brings a theoretical understanding of the principles behind the empirical success of deep neural networks. The synergy between the theoretical statistical physics approach and scientific questions from machine learning enables a leap forward in our understanding of learning from data.
