DeepLearning 2.0: Meta-Learning Qualitatively New Components

Project Information

DeepLearning 2.0

Grant agreement ID: 101045765

DOI

10.3030/101045765

EC signature date 14 April 2022

Start date 1 May 2022

End date 30 April 2027

Funded under

• European Research Council (ERC)

Total cost

€ 2 000 000,00

EU contribution

€ 2 000 000,00

Coordinated by

ALBERT-LUDWIGS-UNIVERSITAET FREIBURG
Germany

Periodic Reporting for period 1 - DeepLearning 2.0 (DeepLearning 2.0: Meta-Learning Qualitatively New Components)

Reporting period: 2022-05-01 to 2024-10-31

Summary of the context and overall objectives of the project

In the expanding realm of artificial intelligence, deep learning has emerged as the key technology behind revolutionary applications across various domains, from interpreting complex images to decoding human language.
Yet, despite its vast achievements, the deep learning we know today stands on the cusp of an evolutionary leap—a transition from hand-engineered problem-solving to sophisticated learned solutions that outsmart hand-engineered ones. The success of deep learning is based on replacing domain-specific handcrafted feature engineering with features that are learned for the particular task at hand. This is an example of the fact that, throughout the history of AI, there is a clear trend in machine learning that hand-crafted solutions are eventually replaced by more effective, learned solutions. The vision behind our ambitious project DEEP LEARNING 2.0 is to repeat the success of deep learning at the meta-level, replacing handcrafted algorithmic components of deep learning by improved, learned ones.

At the heart of our proposed next-gen AI evolution is the concept of meta-learning customized components of deep learning pipelines; a concept where a meta-level AI (meta-)learns improved components of its learning process, molding the neural networks, optimizers, hyperparameters, and training data to fit specific needs and tasks dynamically. Among these transformative possibilities is the potential for AI to recognize and actively integrate multiple user-desired objectives next to predictive accuracy, like robustness, energy usage, latency, interpretability, and fairness. Embracing these user-defined objectives and directly optimizing for them, DEEP LEARNING 2.0 aspires to set the stage for truly Trustworthy AI by design.
The transformative potential of DEEP LEARNING 2.0 will be demonstrated in various applications, such as tabular data, EEG decoding, RNA folding, or reinforcement learning.

The significance of DEEP LEARNING 2.0 pivots not only on advancing the science of AI but also on its promise to democratize deep learning—making it a tool accessible for broader social, academic, and commercial utility. Reflecting its potential to advance the foundation of modern deep learning, the project's impact could ripple through the entire ecosystem of AI, where each breakthrough elevates industry capabilities, advances academic frontiers, and touches lives profoundly with smarter, fairer, and more sustainable AI technology.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

We achieved significant milestones across five main work packages (WPs), each contributing to our overarching goal of advancing deep learning by automating and enhancing its fundamental components:

WP 1: Hierarchical, Efficient, and Multi-objective Neural Architecture Search (NAS)
Progress in WP1 was marked by foundational breakthroughs in multi-objective NAS, including a novel gradient-based NAS approach for weight-entangled spaces like Transformers and the creation of joint NAS+HPO benchmarks. Key publications include several papers at NeurIPS, addressing hierarchical search and multi-objective optimization.

WP 2: Context-Aware Optimizers
Research here focused on the development of a benchmarking framework for neural optimizer search (NOS) and the creation of lightweight benchmarks for evaluating deep learning optimizers.

WP 3: Learning the Data to Train On
Computational constraints limited large-scale experiments, but we made substantial progress in tuning hyperparameters and data augmentations for self-supervised learning. Work on algorithmically generating data was advanced significantly, particularly through TabPFN’s contributions.

WP 4: Bootstrapping from Prior Design Efforts
This WP yielded cutting-edge methods in Bayesian optimization, multi-fidelity optimization, and meta-learning. Notable contributions include the development of JES, a popular acquisition function, and Priorband, which combines user priors with multi-fidelity optimization.

WP 5: Applications
Significant advances were achieved in tabular data (TabPFN), RNA design, and reinforcement learning. TabPFN stands out as a major breakthrough, redefining the landscape of tabular machine learning.

In total, we published 34 papers, including 10 full NeurIPS papers, and conducted significant interdisciplinary research, particularly in RNA folding and design.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The results of DEEP LEARNING 2.0 represent a substantial advancement beyond the current state-of-the-art in artificial intelligence research and its practical applications:

Meta-learning breakthroughs: TabPFN demonstrated the practical viability of meta-learning for tabular data, achieving superior performance with in-context learning. This result exemplifies a shift towards leveraging foundation models to solve traditional machine learning problems.

Enhanced multi-objective optimization: The development of ModNAS and Priorband enables effective optimization across diverse objectives, including accuracy, robustness, and energy efficiency, positioning AI as a tool for Trustworthy AI by design.

New benchmarks and tools: The creation of benchmarks for NAS, NOS, and data-centric AI research provides the scientific community with critical tools for future progress, ensuring the replicability and scalability of foundational deep learning methodologies.

Interdisciplinary impacts: The integration of deep learning with RNA design has opened new frontiers in computational biology, with potential implications for drug discovery and synthetic biology.

Wider societal benefits: By reducing the barrier to entry for complex AI techniques, DEEP LEARNING 2.0 supports democratization, enabling more equitable access to cutting-edge technologies across academia, industry, and beyond.

The long-term impact of this project depends on sustained investment in infrastructure and collaboration. Future research should focus on scaling up these innovations, extending their applicability to new domains, and addressing standardization challenges to ensure safe and ethical deployment. We anticipate that these advancements will not only influence the field of AI but also pave the way for broader societal and economic benefits.