To democratize deep learning and broaden its applicability, the project aims to develop small-scale, resourceefficient models. The overarching goal is to promote sparsity throughout the machine learning pipeline to obtain models that are more scalable, resource-efficient, robust to noise, and better interpretable. The work specifically advances two core challenges: (i) identifying trainable sparse network structures and (ii) enabling de novo optimization of small-scale (sparse) models.
Across six work packages (with WP3 and WP6 as the most ambitious/high-impact targets), we delivered theoretical and algorithmic progress toward sparse architecture design and efficient optimization:
WP1–WP2 (foundations + learning with strong inductive bias):
We developed a graphon–NTK framework for sparse random graph ensembles and analyzed how masking affects conditioning and thus trainability. Novel balance equations that connect gradients and parameter dynamics inform better initializations, architectures, and optimization dynamics. We also advanced parameterand memory-efficient fine-tuning by leveraging inductive bias in pre-trained models and showed that selectively updating parameters (or even freezing message passing in some GNN settings) can improve generalization.
WP3 (macroscopic learning):
We pursued learning parameter distributions on a macro level, supported by convergence results, while working
toward more efficient practical algorithms. A key conceptual step was linking overparameterization to OOD
robustness via hidden confounding, motivating more efficient confounder-identification strategies and implicitbias
control through regularization.
WP4–WP5 (structure design):
We designed domain-informed sparse structures for a NeuralODE approach to gene regulatory dynamics that
scales to the full human genome and improved pruning using domain knowledge to boost efficiency and interpretability.
In parallel, we advanced sparsification via distillation and identified an extremely simple criterion for
PEFT masking in language models (i.e. small weights), while emphasizing the need to reduce dense training
whenever possible.
WP6 (de novo sparse optimization):
We identified sign learning as a central bottleneck in sparse training and tied it to unfavorable implicit bias near
the origin. This led to methods for implicit-bias control by explicit regularization, the Sign-In approach, and
the recent optimizer HAM, which extends first-order methods with an exponential update and yields strong
performance gains even for dense training.
Beyond the technical results, the project enabled visible knowledge transfer and community impact and sparked
industry collaboration interest related to efficient LLMs.
