Unified Theory of Efficient Optimization and Estimation

Informazioni relative al progetto

UTOPEST

ID dell’accordo di sovvenzione: 815464

DOI

10.3030/815464

Progetto chiuso

Data della firma CE 19 Dicembre 2018

Data di avvio 1 Marzo 2019

Data di completamento 31 Agosto 2024

Finanziato da

• EXCELLENT SCIENCE - European Research Council (ERC)

Costo totale

€ 1 993 320,00

Contributo UE

€ 1 993 320,00

Coordinato da

EIDGENOESSISCHE TECHNISCHE HOCHSCHULE ZUERICH
Switzerland

Periodic Reporting for period 4 - UTOPEST (Unified Theory of Efficient Optimization and Estimation)

Periodo di rendicontazione: 2023-09-01 al 2024-08-31

Sintesi del contesto e degli obiettivi generali del progetto

Optimization and estimation are the computational engines driving modern data science and artificial intelligence. From training neural networks to analyzing social networks, these mathematical problems are ubiquitous.
However, a significant gap often exists between what is information-theoretically achievable and what can be computed efficiently by current algorithms.
Furthermore, standard algorithms are frequently "fragile": they perform well on idealized data but fail catastrophically when data is noisy, heavy-tailed, or manipulated by malicious adversaries—a scenario increasingly common in real-world applications.

The UTOPEST project aimed to bridge this gap by developing a unified algorithmic theory for efficient, robust optimization and estimation.
Our central hypothesis was that the Sum-of-Squares (SoS) method—a powerful meta-algorithm based on semidefinite programming—could serve as a canonical framework to achieve the best possible provable guarantees.

Overall Objectives
The primary scientific objective was to determine the limits of efficient computation for high-dimensional estimation.
Specifically, we sought to:
- Develop efficient algorithms that match information-theoretic limits for robust estimation and clustering.
- Understand the "price of robustness": does requiring an algorithm to be robust necessitate more data or computation?
- Establish a unified theory that treats convex and non-convex optimization, as well as robust and non-robust estimation, under a single framework.

Conclusions of the Action
Over the course of the project, we successfully demonstrated that the Sum-of-Squares method indeed provides this unified framework.
- Robustness at no cost: We proved that for many fundamental problems (such as community detection and sparse PCA), it is possible to achieve robustness against adversarial corruption without sacrificing statistical accuracy or computational efficiency.
- Universality: We resolved a major open question by showing that general "subgaussian" distributions (a broad class of realistic data distributions) can be handled as efficiently as ideal Gaussian data, effectively unifying the theory for a vast range of statistical tasks.
- Privacy: In a significant expansion of our original scope, we established a deep algorithmic connection between robustness and Differential Privacy. We used our robust optimization techniques to design the first efficient algorithms that preserve privacy in complex network analysis, resolving a long-standing trade-off between privacy and efficiency.

Lavoro eseguito dall’inizio del progetto fino alla fine del periodo coperto dalla relazione e principali risultati finora ottenuti

A central theme of our work was determining whether "robustness"—the ability of an algorithm to function correctly even when data is manipulated by an adversary—comes at a cost.
- In the domain of unsupervised learning, we analyzed fundamental problems like Sparse Principal Component Analysis (FOCS 2020) and Community Detection (FOCS 2021). We demonstrated that standard algorithms are indeed fragile, failing under even minor corruptions. However, we proved that using our Sum-of-Squares techniques, it is possible to achieve robustness against significant corruption without requiring more data or computation than the fragile baselines.
- This line of research culminated in a breakthrough result (FOCS 2025) showing that general subgaussian distributions—a broad class of realistic data models—can be handled as efficiently as idealized Gaussian data. This effectively unifies the theory of robust estimation for a vast range of real-world scenarios.

In a significant expansion of the project's original scope, we established a deep connection between robust statistics and Differential Privacy—the gold standard for protecting individual data.
- We showed that the mathematical "certificates" we developed to detect adversarial corruption could also be used to mask the influence of any single individual in a dataset.
- This led to the design of the first efficient algorithms for analyzing sensitive network data (e.g. social graphs) that simultaneously guarantee strict privacy and optimal statistical accuracy (NeurIPS 2024, STOC 2024).

The results of the UTOPEST project have been disseminated widely within the scientific community:
- Publications: The project generated over 30 peer-reviewed papers published in the top-tier conferences of the field (STOC, FOCS, SODA, NeurIPS, COLT), ensuring high visibility among experts in algorithms and machine learning.
- Awards: The quality of this research was recognized internationally, including a Best Student Paper Award at COLT 2024.
- Training: The project served as a training hub for the next generation of researchers. Team members have successfully transitioned to tenure-track professorships (e.g. at Bocconi University) and permanent research positions in industry, and have secured their own competitive research funding (e.g. NWO Veni grant).

Progressi oltre lo stato dell’arte e potenziale impatto previsto (incluso l’impatto socioeconomico e le implicazioni sociali più ampie del progetto fino ad ora)

The UTOPEST project has fundamentally advanced the field of algorithmic statistics by establishing a unified theory for efficient, robust, and private estimation. At the project's inception, the state of the art was characterized by fragmented techniques and sharp trade-offs between computational efficiency, statistical accuracy, and robustness. We have successfully moved beyond these limitations in three key areas:

Prior to this project, efficient estimation algorithms were typically "fragile"—failing catastrophically under heavy-tailed noise or adversarial data corruption. Conversely, robust methods were often computationally intractable (requiring exponential time).
We resolved this dichotomy by proving that the Sum-of-Squares (SoS) hierarchy provides a universal framework for robust efficiency.
A crowning achievement of the project is the resolution of the long-standing "subgaussian gap" (FOCS 2025). We proved that general, realistic data distributions (subgaussian) can be handled with the same computational efficiency and optimal error rates as idealized Gaussian data. This result effectively generalizes the theory of efficient robust estimation to a vast class of real-world distributions, moving the field beyond the restrictive Gaussian assumptions that previously dominated the literature.

In the domain of Differential Privacy, particularly for complex graph data, the state of the art suggested an inherent conflict: algorithms could be computationally efficient or statistically optimal, but not both.
Our research removed this barrier. By establishing a novel algorithmic connection between robust statistics and privacy, we developed the first polynomial-time algorithms for private network analysis (e.g. Graphon and Edge Density estimation) that achieve information-theoretic optimal accuracy.
This progress demonstrates that protecting individual privacy in social network analysis does not require a compromise on the quality of insights or the use of prohibitive computational resources.

The project has significantly refined our understanding of the limits of efficient computation.
- Semirandom Models: We showed that for fundamental problems like Planted Clique, robustness against "semi-random" adversaries comes at no cost to the signal-to-noise threshold.
- Hardness of Learning: We provided the tightest evidence to date for the intractability of learning intersections of halfspaces, narrowing the gap between upper and lower bounds to a theoretical sliver.

semi-random planted clique problem