Periodic Reporting for period 1 - CODY (The Complexity of Dynamic Matrix Problems)
Reporting period: 2022-07-01 to 2024-12-31
Modern data analysis and optimization rely on the ability to efficiently process high-dimensional dynamic datasets, where data is rapidly evolving.
Dynamic data structures enable fast information-retrieval and algebraic operations on these huge databases by maintaining implicit information on
the underlying data. Understanding the power and limitations of dynamic data structure, and the role they play in accelerating large-scale algorithms,
is therefore a fundamental question in theory and practice.
The main objective of this project is: (1) Making progress on fundamental theoretical problems in dynamic data structures, such as proving lower bounds
for the computational cost of dynamically maintaining network connectivity. (2) Designing dynamic data data structures for accelerating continuous optimization
(from Linear Programming to Deep Learning): The last few years showcased the power of dynamic data structures in reducing the cost-per-iteration of (Newton type)
optimization algorithms, proclaiming that the bottleneck to accelerating literally thousands of algorithms, is efficient maintenance of dynamic linear equations and
inverse operators.
This proposal lays out new technical and conceptual directions for addressing some of the most fundamental problems in data structures and optimization,
and has a far-reaching potential impact on algorithm design and large-scale optimization.
Dynamic data structures enable fast information-retrieval and algebraic operations on these huge databases by maintaining implicit information on
the underlying data. Understanding the power and limitations of dynamic data structure, and the role they play in accelerating large-scale algorithms,
is therefore a fundamental question in theory and practice.
The main objective of this project is: (1) Making progress on fundamental theoretical problems in dynamic data structures, such as proving lower bounds
for the computational cost of dynamically maintaining network connectivity. (2) Designing dynamic data data structures for accelerating continuous optimization
(from Linear Programming to Deep Learning): The last few years showcased the power of dynamic data structures in reducing the cost-per-iteration of (Newton type)
optimization algorithms, proclaiming that the bottleneck to accelerating literally thousands of algorithms, is efficient maintenance of dynamic linear equations and
inverse operators.
This proposal lays out new technical and conceptual directions for addressing some of the most fundamental problems in data structures and optimization,
and has a far-reaching potential impact on algorithm design and large-scale optimization.
Summary of Major Achievements:
In the two years since the inception of this STG project, me and my research group have made a substantial progress on the following four
problems mentioned in the CODY project proposal:
•Problem 2: How can dynamic data structures and sketching reduce the cost-per-iteration of IPMs and other path-following homotopy methods?
•Problem 3: Are Linear Programs and Linear systems computationally equivalent?
•Problem 4 : Can we prove non-black-box lower bounds on the iteration complexity of interior point methods (IPMs)?
•Problem 5 : How can we design practical algorithms for computing high-accuracy approximate matrix products in close to O(n^2) (or ideally matrix sparsity) time?
For Problems 2 and 4, my research group has developed novel upper and lower bounds on the iteration complexity and cost-per-iteration of Interior Point Methods
(IPMS) for Linear and Semidefinite Programming (items 5+6 and papers [2],[8] below), perhaps the most important question of this Grant project. Another set of
main achievements of this reporting period (addressing Problem 3 and 5) include novel dynamic data structures for estimating distances of general symmetric norms
(paper [2] below), for sparse recovery of Fourier-spare signals (paper [3]), and for dynamic least-squares regression (paper [6], resolving the complexity of a
fundamental problem in data analysis);
**UPDATE** : Papers [9] + [10] accepted to STOC'25 .
*UPCOMING Papers and work in progress:
*For Problem 5: In my upcoming (solo-author) paper on ”Approximate Matrix Multiplication via Spherical Convolutions” (under preparation), I present a cheaper
(quadratic instead of cubic-time) Bilinear operator as an alternative to MatMul in deep neural networks, which preserves the number of parameters of the model,
providing an end-to-end speedup of up to x3 in FLOPs over naiive MatMul.
*In an upcoming paper with an ERC Postdoc (Item [4] below, to be submitted to STOC'25), we have developed the first accelerated multiplicative-weights method
for approximate packing/covering LPs after nearly 20 years of research (Garg-Konneman,2007), providing quadratic speed up in the number of iterations (m --> m^{1/2})
using a different oracle. I was invited to give this preliminary track at the Simons Institute Data Structures & Optimization Semester (Berkeley, Nov'23).
* In another upcoming project with an ERC Postdoc (Item [7] below), we have developed a novel approach To the longstanding Dynamic Optimzality Conjecture
(Tarjan & Sleator, 1985), based on a continuous relaxation -- We have managed to encode the execution of any online binary search tree on a sequence x1...xm of lookups,
as an (implicit) online flow-based LP. Using This formulation provides an approach for attacking the Dynamic Optimality conjecture via online convex-body chasing,
and designing An entire new class of BST algorithms based on regularized gradient descent (online Bergman projections).
[1] [Y. Deng, Z.Song , O.WeinsteinC R.Zhang] Fast Distance Oracles for Any Symmetric Norm. (2022). NeurIPS23.
[2] [S.Jiang B.Peng O.Weinstein] Dynamic Least-Squares regression. 2023 IEEE Symposium on Foundations of Computer Science (FOCS23).
[3] [Z. Song, B. Sun, O. Weinstein, R. Zhang] Quartic Samples Suffice for Fourier Interpolation. 2023 IEEE Symposium on Foundations of Computer Science (FOCS23).
[4] [Z. Kuhn-Koh, O.Weinstein S.Yincharantawanchai]: An Accelerated Multiplicative-Weights Update Method for Implicit Packing-Covering LPs (to be submitted to STOC'25)
[5] [S.Jiang D. Ming, R.Kyng O.Weinstein]: How Many Linear Systems are Required to Solve a Linear Program? (To be submitted to ITCS'24)
[6] [S.Jiang A.Vladu O.Weinstein]: IPMs Beyond Self-Concordance via Energy-Based Perturbations (In preparation).
[7] [D.Dorfman P.Kalmansook O.Weinstein S.Yincharantawanchai]: Dynamic Optimality via Continuous Optimization (work in progress).
[8] [S.Jiang B. Natura , O.Weinstein]: “A Faster Interior-Point Method for Sum of Squares Optimization” (ICALP'22)
[9] [Z. Kuhn-Koh, O.Weinstein S.Yincharantawanchai]: Approximating the Held–Karp Bound of Metric TSP in Nearly-Linear Work and Polylogarithmic Depth (accepted to STOC'25)
[10] [A. Andoni, S. Jiang, O.Weinstein]: "A New Framework for Building Data Structures from Communication Protocols" (accepted to STOC'25)
In the two years since the inception of this STG project, me and my research group have made a substantial progress on the following four
problems mentioned in the CODY project proposal:
•Problem 2: How can dynamic data structures and sketching reduce the cost-per-iteration of IPMs and other path-following homotopy methods?
•Problem 3: Are Linear Programs and Linear systems computationally equivalent?
•Problem 4 : Can we prove non-black-box lower bounds on the iteration complexity of interior point methods (IPMs)?
•Problem 5 : How can we design practical algorithms for computing high-accuracy approximate matrix products in close to O(n^2) (or ideally matrix sparsity) time?
For Problems 2 and 4, my research group has developed novel upper and lower bounds on the iteration complexity and cost-per-iteration of Interior Point Methods
(IPMS) for Linear and Semidefinite Programming (items 5+6 and papers [2],[8] below), perhaps the most important question of this Grant project. Another set of
main achievements of this reporting period (addressing Problem 3 and 5) include novel dynamic data structures for estimating distances of general symmetric norms
(paper [2] below), for sparse recovery of Fourier-spare signals (paper [3]), and for dynamic least-squares regression (paper [6], resolving the complexity of a
fundamental problem in data analysis);
**UPDATE** : Papers [9] + [10] accepted to STOC'25 .
*UPCOMING Papers and work in progress:
*For Problem 5: In my upcoming (solo-author) paper on ”Approximate Matrix Multiplication via Spherical Convolutions” (under preparation), I present a cheaper
(quadratic instead of cubic-time) Bilinear operator as an alternative to MatMul in deep neural networks, which preserves the number of parameters of the model,
providing an end-to-end speedup of up to x3 in FLOPs over naiive MatMul.
*In an upcoming paper with an ERC Postdoc (Item [4] below, to be submitted to STOC'25), we have developed the first accelerated multiplicative-weights method
for approximate packing/covering LPs after nearly 20 years of research (Garg-Konneman,2007), providing quadratic speed up in the number of iterations (m --> m^{1/2})
using a different oracle. I was invited to give this preliminary track at the Simons Institute Data Structures & Optimization Semester (Berkeley, Nov'23).
* In another upcoming project with an ERC Postdoc (Item [7] below), we have developed a novel approach To the longstanding Dynamic Optimzality Conjecture
(Tarjan & Sleator, 1985), based on a continuous relaxation -- We have managed to encode the execution of any online binary search tree on a sequence x1...xm of lookups,
as an (implicit) online flow-based LP. Using This formulation provides an approach for attacking the Dynamic Optimality conjecture via online convex-body chasing,
and designing An entire new class of BST algorithms based on regularized gradient descent (online Bergman projections).
[1] [Y. Deng, Z.Song , O.WeinsteinC R.Zhang] Fast Distance Oracles for Any Symmetric Norm. (2022). NeurIPS23.
[2] [S.Jiang B.Peng O.Weinstein] Dynamic Least-Squares regression. 2023 IEEE Symposium on Foundations of Computer Science (FOCS23).
[3] [Z. Song, B. Sun, O. Weinstein, R. Zhang] Quartic Samples Suffice for Fourier Interpolation. 2023 IEEE Symposium on Foundations of Computer Science (FOCS23).
[4] [Z. Kuhn-Koh, O.Weinstein S.Yincharantawanchai]: An Accelerated Multiplicative-Weights Update Method for Implicit Packing-Covering LPs (to be submitted to STOC'25)
[5] [S.Jiang D. Ming, R.Kyng O.Weinstein]: How Many Linear Systems are Required to Solve a Linear Program? (To be submitted to ITCS'24)
[6] [S.Jiang A.Vladu O.Weinstein]: IPMs Beyond Self-Concordance via Energy-Based Perturbations (In preparation).
[7] [D.Dorfman P.Kalmansook O.Weinstein S.Yincharantawanchai]: Dynamic Optimality via Continuous Optimization (work in progress).
[8] [S.Jiang B. Natura , O.Weinstein]: “A Faster Interior-Point Method for Sum of Squares Optimization” (ICALP'22)
[9] [Z. Kuhn-Koh, O.Weinstein S.Yincharantawanchai]: Approximating the Held–Karp Bound of Metric TSP in Nearly-Linear Work and Polylogarithmic Depth (accepted to STOC'25)
[10] [A. Andoni, S. Jiang, O.Weinstein]: "A New Framework for Building Data Structures from Communication Protocols" (accepted to STOC'25)
Our two FOCS'23 papers mentioned above, essentially settle the complexity of two very fundamental problems
-- Dynamic least squares regression and Recovery of Fourier-Sparse Signals. The upcoming papers [4] demonstrate
the *first* sublinear iteration algoriothm for low-accuracy Linear Programming, breaking the classical result of
[Garg-Konnemann, 1998] and [Plotkin, Shmoys, Tardos, 91]. Paper [9] provides the first *parallel* (polylogarithmic
depth) algorithm with near-*linear* work ~O(m) for Metric TSP (traveling salesman problem), one of the cornerstones
of computer science and algorithm design. This is done via a novel *combinatorial* insight about pattern-avoiding
structure of approximate minimum cuts, which is of independent interest.
Notwithstanding, I believe the two most significant achievemnents of this ERC project are not yet published:
(I) The continuous (Flow LP) formulation of Dynamic Optimality Conjecture mentioned in the above section "Work performed
and main achievements", which seems like a real breakthrough in the area. This project is still under research, but we already
have promising results about this fundamentally new approach.
(II) My upcoming paper on Approximate Matrix Multiplication via Spherical Convolutions (still under preparation), which demonstrates
a cheaper alternative to MatMul in deep neural networks, *without decreasing the number of parameters of the model* -- providing an
end-to-end speedup of up to x3 in FLOPs over naiive MatMul for typical layers in state of art Transformers.
-- Dynamic least squares regression and Recovery of Fourier-Sparse Signals. The upcoming papers [4] demonstrate
the *first* sublinear iteration algoriothm for low-accuracy Linear Programming, breaking the classical result of
[Garg-Konnemann, 1998] and [Plotkin, Shmoys, Tardos, 91]. Paper [9] provides the first *parallel* (polylogarithmic
depth) algorithm with near-*linear* work ~O(m) for Metric TSP (traveling salesman problem), one of the cornerstones
of computer science and algorithm design. This is done via a novel *combinatorial* insight about pattern-avoiding
structure of approximate minimum cuts, which is of independent interest.
Notwithstanding, I believe the two most significant achievemnents of this ERC project are not yet published:
(I) The continuous (Flow LP) formulation of Dynamic Optimality Conjecture mentioned in the above section "Work performed
and main achievements", which seems like a real breakthrough in the area. This project is still under research, but we already
have promising results about this fundamentally new approach.
(II) My upcoming paper on Approximate Matrix Multiplication via Spherical Convolutions (still under preparation), which demonstrates
a cheaper alternative to MatMul in deep neural networks, *without decreasing the number of parameters of the model* -- providing an
end-to-end speedup of up to x3 in FLOPs over naiive MatMul for typical layers in state of art Transformers.