Objective
Modern machine learning models have been successfully deployed across fields, from scientific studies to tech-
nological developments in industry, but their development remains poorly understood. The training of a large
language model such as GPT-3 is estimated to cost $4.6M and public attempts to replicate the training process
alone required teams of engineers to rotating on-call for months, monitoring various statistics and constantly
tweaking the training procedure when it broke. Existing theoretical frameworks offer limited insights into this
process, as they do not capture the main difficulties that arise in practice when training neural networks, leaving
practitioners to rely on error-prone heuristics and expensive trial-and-error. This leads not only to a large devel-
opment cost dominated by wasted resources, but also limits the possible impacts of machine learning to areas
considered profitable by industries that have the resources to carry this development.
The objective of this project is to build a better understanding of how recently identified bottlenecks in neural
network training slow down optimization and how to adress them. The specific aims are to: (a) Understand
the impact of class imbalance on the dynamics of neural networks to identify where to allocate algorithmic
resources. (b) Develop a theory to capture optimization difficulties early in training to guide the development
of algorithms that improve performance during this crucial phase. (c) Identify new bottlenecks that arise from
applications to new data types.
The project combines experimental expertise of the postdoctoral and the theoretical expertise of the host insti-
tution to identify and describe the real impact of data characteristics on neural network training. Understanding
these bottlenecks will help develop more efficient and reliable algorithms and guidelines on best practices that
depend on properties of the data.
Fields of science (EuroSciVoc)
CORDIS classifies projects with EuroSciVoc, a multilingual taxonomy of fields of science, through a semi-automatic process based on NLP techniques.
CORDIS classifies projects with EuroSciVoc, a multilingual taxonomy of fields of science, through a semi-automatic process based on NLP techniques.
You need to log in or register to use this function
Programme(s)
- HORIZON.1.2 - Marie Skłodowska-Curie Actions (MSCA) Main Programme
Funding Scheme
HORIZON-TMA-MSCA-PF-EF - HORIZON TMA MSCA Postdoctoral Fellowships - European FellowshipsCoordinator
78153 Le Chesnay Cedex
France