Distributed Optimization Beyond Worst-Case Topologies

Project Information

DistOpt-BydWorstCase

Grant agreement ID: 949272

DOI

10.3030/949272

Project closed

EC signature date 17 August 2020

Start date 1 September 2020

End date 31 January 2026

Funded under

• EXCELLENT SCIENCE - European Research Council (ERC)

Total cost

€ 1 540 000,00

EU contribution

€ 1 540 000,00

Coordinated by

EIDGENOESSISCHE TECHNISCHE HOCHSCHULE ZUERICH
Switzerland

Periodic Reporting for period 3 - DistOpt-BydWorstCase (Distributed Optimization Beyond Worst-Case Topologies)

Reporting period: 2023-09-01 to 2025-02-28

Summary of the context and overall objectives of the project

Modern computer systems are increasingly decentralized and massively distributed computations play a vital role in the systems of the future. This project aims to advance the foundational aspects of distributed computing, particularly MessagePassing algorithms for optimization problems. The strong shift towards distributed systems also lead to a flurry of works on such algorithms -- many optimization problems now have distributed algorithms with optimal worst-case performance guarantees. Generally, these guarantees cannot be improved because they match unconditional impossibility results, which prove severe limitations on the performance of distributed algorithms in some (pathological) network topologies. Real world networks, however, are never worst-case and do not share the limiting bottleneck characteristics of these pathological topologies. In fact, there is no known barrier for ultra-fast polylogarithmic-round distributed algorithms on any network of interest. This leaves an exponential gap between current worst-case-optimal algorithms and what is likely possible in many, if not all, real-world settings.
Motivated by this, this project pursues a program for a general toolbox and theory for MessagePassing optimization algorithms that go beyond worst-case topologies. The main and guiding high-risk high-gain goal is the development of universally optimal distributed algorithms, which are competitive with the best algorithm on any given topology. This would constitute the strongest possible form of algorithmically adjusting to non-worst-case topologies and contribute to and strengthen the (theoretical) foundation to the modern and future distributed systems society relies upon. The project follows a detailed program with many concrete and smaller stepping stones towards this ambitious breakthrough objective. Many of the novel questions stemming from this program and proposal are highly interdisciplinary, crossing boundaries between information theory, distributed computing, topological graph theory, and other parts of theoretical computer science, and are fundamental in their own right.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

Work on this project begun immediately with the starting date of the award in September 2020 and the PI has spent the vast majority of their efforts on this project. COVID-19 made in-person collaborations and the recruitment of personnel to this project initially challenging but the project managed to recruit several M.Sc. and PhD students and one postdocs to ETHZ for this project. The PI spent substantial efforts beyond the anticipated 65% on the project. Significant research results beyond the state of the art were obtained as a result of this effort and published in highly-competitive peer-reviewed venues. The PI and other team members also widely disseminated the research results obtained and the general research direction set forth by this project in various talks at conferences, workshops, and invited talks at universities and industrial research labs. This includes keynotes by the PI at ALGO 2021, the International Conference on Distributed Computing (DISC 2021), and an invited spot-light talk at the Workshop on Local Algorithms (WOLA) and invitation to speak during the Shonan Workshop on "New Frontiers in Locality in Computation" in Japan.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The project let to the first universally optimal distributed algorithms in the known topology setting and to break-through results in hop-constrained oblivious routing which are tightly linked to efficient distributed constructions of low-congestion shortcuts. New and powerful tree-embeddings for hop-constrained distances needed to be developed to construct these oblivious routings. These (partial) tree-embeddings have led to significant further results in hop-constrained versions of classical online and approximation algorithms for network design. These significant results beyond the state of art have let to several dozens publications at the top venues in the area (STOC, FOCS, SODA, PODC, etc). Further publications are in preparation and submission. Progress on this project has been so successful that it is expected that even the most ambitious goals outlined in the proposal might be achievable until the end of the project. Further efforts and results building on these new foundations as well as work on implications and applications of the newly developed tools are already in progress.

picture.jpg