Unified Theory of Efficient Optimization and Estimation

Projektinformationen

UTOPEST

ID Finanzhilfevereinbarung: 815464

DOI

10.3030/815464

Projekt abgeschlossen

EK-Unterschriftsdatum 19 Dezember 2018

Startdatum 1 März 2019

Enddatum 31 August 2024

Finanziert unter

• EXCELLENT SCIENCE - European Research Council (ERC)

Gesamtkosten

€ 1 993 320,00

EU-Beitrag

€ 1 993 320,00

Koordiniert durch

EIDGENOESSISCHE TECHNISCHE HOCHSCHULE ZUERICH
Switzerland

Periodic Reporting for period 4 - UTOPEST (Unified Theory of Efficient Optimization and Estimation)

Berichtszeitraum: 2023-09-01 bis 2024-08-31

Zusammenfassung vom Kontext und den Gesamtzielen des Projekts

Optimization and estimation are the computational engines driving modern data science and artificial intelligence. From training neural networks to analyzing social networks, these mathematical problems are ubiquitous.
However, a significant gap often exists between what is information-theoretically achievable and what can be computed efficiently by current algorithms.
Furthermore, standard algorithms are frequently "fragile": they perform well on idealized data but fail catastrophically when data is noisy, heavy-tailed, or manipulated by malicious adversaries—a scenario increasingly common in real-world applications.

The UTOPEST project aimed to bridge this gap by developing a unified algorithmic theory for efficient, robust optimization and estimation.
Our central hypothesis was that the Sum-of-Squares (SoS) method—a powerful meta-algorithm based on semidefinite programming—could serve as a canonical framework to achieve the best possible provable guarantees.

Overall Objectives
The primary scientific objective was to determine the limits of efficient computation for high-dimensional estimation.
Specifically, we sought to:
- Develop efficient algorithms that match information-theoretic limits for robust estimation and clustering.
- Understand the "price of robustness": does requiring an algorithm to be robust necessitate more data or computation?
- Establish a unified theory that treats convex and non-convex optimization, as well as robust and non-robust estimation, under a single framework.

Conclusions of the Action
Over the course of the project, we successfully demonstrated that the Sum-of-Squares method indeed provides this unified framework.
- Robustness at no cost: We proved that for many fundamental problems (such as community detection and sparse PCA), it is possible to achieve robustness against adversarial corruption without sacrificing statistical accuracy or computational efficiency.
- Universality: We resolved a major open question by showing that general "subgaussian" distributions (a broad class of realistic data distributions) can be handled as efficiently as ideal Gaussian data, effectively unifying the theory for a vast range of statistical tasks.
- Privacy: In a significant expansion of our original scope, we established a deep algorithmic connection between robustness and Differential Privacy. We used our robust optimization techniques to design the first efficient algorithms that preserve privacy in complex network analysis, resolving a long-standing trade-off between privacy and efficiency.

Arbeit, die ab Beginn des Projekts bis zum Ende des durch den Bericht erfassten Berichtszeitraums geleistet wurde, und die wichtigsten bis dahin erzielten Ergebnisse

A central theme of our work was determining whether "robustness"—the ability of an algorithm to function correctly even when data is manipulated by an adversary—comes at a cost.
- In the domain of unsupervised learning, we analyzed fundamental problems like Sparse Principal Component Analysis (FOCS 2020) and Community Detection (FOCS 2021). We demonstrated that standard algorithms are indeed fragile, failing under even minor corruptions. However, we proved that using our Sum-of-Squares techniques, it is possible to achieve robustness against significant corruption without requiring more data or computation than the fragile baselines.
- This line of research culminated in a breakthrough result (FOCS 2025) showing that general subgaussian distributions—a broad class of realistic data models—can be handled as efficiently as idealized Gaussian data. This effectively unifies the theory of robust estimation for a vast range of real-world scenarios.

In a significant expansion of the project's original scope, we established a deep connection between robust statistics and Differential Privacy—the gold standard for protecting individual data.
- We showed that the mathematical "certificates" we developed to detect adversarial corruption could also be used to mask the influence of any single individual in a dataset.
- This led to the design of the first efficient algorithms for analyzing sensitive network data (e.g. social graphs) that simultaneously guarantee strict privacy and optimal statistical accuracy (NeurIPS 2024, STOC 2024).

The results of the UTOPEST project have been disseminated widely within the scientific community:
- Publications: The project generated over 30 peer-reviewed papers published in the top-tier conferences of the field (STOC, FOCS, SODA, NeurIPS, COLT), ensuring high visibility among experts in algorithms and machine learning.
- Awards: The quality of this research was recognized internationally, including a Best Student Paper Award at COLT 2024.
- Training: The project served as a training hub for the next generation of researchers. Team members have successfully transitioned to tenure-track professorships (e.g. at Bocconi University) and permanent research positions in industry, and have secured their own competitive research funding (e.g. NWO Veni grant).

Fortschritte, die über den aktuellen Stand der Technik hinausgehen und voraussichtliche potenzielle Auswirkungen (einschließlich der bis dato erzielten sozioökonomischen Auswirkungen und weiter gefassten gesellschaftlichen Auswirkungen des Projekts)

The UTOPEST project has fundamentally advanced the field of algorithmic statistics by establishing a unified theory for efficient, robust, and private estimation. At the project's inception, the state of the art was characterized by fragmented techniques and sharp trade-offs between computational efficiency, statistical accuracy, and robustness. We have successfully moved beyond these limitations in three key areas:

Prior to this project, efficient estimation algorithms were typically "fragile"—failing catastrophically under heavy-tailed noise or adversarial data corruption. Conversely, robust methods were often computationally intractable (requiring exponential time).
We resolved this dichotomy by proving that the Sum-of-Squares (SoS) hierarchy provides a universal framework for robust efficiency.
A crowning achievement of the project is the resolution of the long-standing "subgaussian gap" (FOCS 2025). We proved that general, realistic data distributions (subgaussian) can be handled with the same computational efficiency and optimal error rates as idealized Gaussian data. This result effectively generalizes the theory of efficient robust estimation to a vast class of real-world distributions, moving the field beyond the restrictive Gaussian assumptions that previously dominated the literature.

In the domain of Differential Privacy, particularly for complex graph data, the state of the art suggested an inherent conflict: algorithms could be computationally efficient or statistically optimal, but not both.
Our research removed this barrier. By establishing a novel algorithmic connection between robust statistics and privacy, we developed the first polynomial-time algorithms for private network analysis (e.g. Graphon and Edge Density estimation) that achieve information-theoretic optimal accuracy.
This progress demonstrates that protecting individual privacy in social network analysis does not require a compromise on the quality of insights or the use of prohibitive computational resources.

The project has significantly refined our understanding of the limits of efficient computation.
- Semirandom Models: We showed that for fundamental problems like Planted Clique, robustness against "semi-random" adversaries comes at no cost to the signal-to-noise threshold.
- Hardness of Learning: We provided the tightest evidence to date for the intractability of learning intersections of halfspaces, narrowing the gap between upper and lower bounds to a theoretical sliver.

semi-random planted clique problem