The Hardness of Finding Good Algorithms

Project Information

HOFGA

Grant agreement ID: 101041696

DOI

10.3030/101041696

EC signature date 4 August 2022

Start date 1 March 2023

End date 29 February 2028

Funded under

• European Research Council (ERC)

Total cost

€ 1 498 664,00

EU contribution

€ 1 498 664,00

Coordinated by

FACULDADE DE CIENCIAS DA UNIVERSIDADE DE LISBOA
Portugal

Periodic Reporting for period 1 - HOFGA (The Hardness of Finding Good Algorithms)

Reporting period: 2023-03-01 to 2025-08-31

Summary of the context and overall objectives of the project

There are a few fundamental problems at the intersection of Mathematics and Computer Science which we have been trying to solve since the 1950s, during the dawn of computing. These fundamental problems relate with showing that computers are uncapable of solving certain tasks efficiently.

So, for example, if you never heard about these things, it might surprise you to know that cryptography, which we use every day to communicate securely on the internet, is believed to be secure, but we are not at all certain that it actually is. We would like to show, then, that the cryptographic systems we use are secure. By "show", we mean "prove mathematically", i.e. give a logically sound, irrefutable argument showing that no algorithm, no matter how smart, can efficiently decypher encrypted communication.

However, we have been trying to show such impossibility results for many decades, and have systematically failed at doing so. We have failed even at proving much simpler variants of such impossibility results, which in the area we call "lower bounds". The difficulty appears to be that there is a self-referential flavor to the statements we want to prove. In some sense (which can be precisely defined), proving that an algorithm cannot do something, i.e. proving lower bounds, is a task similar to breaking a cryptographic protocol. So there is an apparent contradiction in the task of proving that an algorithm cannot break cryptographic protocols.

It is still very unclear, however, to what extent this contradiction is real or apparent. The goal of this project is to understand this better. There is a number of research directions which relate lower bounds to "automatic programming", which one could call the task of, when given the specification of a computational problem, to obtain an efficient algorithm for solving it.

It turns out that if "automatic programming" is easy enough, it should be possible to prove the kind of impossibility results which we want to prove. And if "automatic programming" is hard enough, secure cryptography is possible. The goal of the project is to study and understand the difficulty of "automatic programming" in several different settings.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

We have studied the difficulty of "automatic programming" in a few different settings:

* Decision trees are one of the simplest type of algorithm which can be studied. We have precisely characterized the difficulty of finding optimal decision trees (it is either NC1-complete or PSPACE-complete, depending on how the problem is encoded).
* Many scenarios in computer science can be moddeled as two parties communicating according to some protocol. We have proven that finding optimal communication protocols is NP-hard.
* An important area in computer science is the design and analysis of data structures for solving various computational tasks. There is a very significant gap between the best known data structures for some problems and the best lower-bounds that we can prove. We have begun work relating this gap with the natural proofs barrier, and this will allow us to understand why this gap is so difficult to overcome.
* We have shown that a particular family of algorithms, semidefinite programming, cannot be used to understand the computational complexity of Boolean circuits and Formulas, and hence cannot be used to find optimal algorithms in these models.

We have also studied a particular family of algorithm for solving stochastic games. Although this was our original purpose, we could not relate these algorithms with automatic programming, but, unexpectedly, we have managed to provide a new analysis of policy iteration - a famous algorithm for this family of problems. Our analysis provides an explanation why policy iteration works well in practice.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

It is the usual nature of mathematical work that new results require understanding new mathematics, and as such all of the work we performed went beyond the state of the art. Of particular note are:

* Our proof that communication complexity is NP-hard comes close to solving a conjecture of Andrew Yao, which he made in the very same paper where communication complexity was invented.
* Our proof that policy iteration runs in smoothed polynomial time solves a longstanding conjecture in the area of stochastic games.