Periodic Reporting for period 4 - SpeedInfTradeoff (Speed-Information Tradeoffs: Beyond Quasi-Entropy Analysis)
Okres sprawozdawczy: 2020-12-01 do 2022-05-31
The main problems being addressed by the project is to obtain lower bounds for computation of the Fourier Transform, using a reasonably solid model of computation. It is widely conjectured that the n log(n) running time of the FFT (Fast Fourier Transform) algorithm is optimal, and the goal of this project is to prove this conjecture, and en route to develop tools for this mathematical analysis that might be useful for other problems.
The project is important for society, because the Fourier transform is probably the most important linear transformation used in area such as signal processing, audio/image/video compressing and more. It is used in big polynomial and integer multiplication (core building blocks of modern crypto) and also in machine learning, especially for deep networks that are based on the convolution operator (Convolutional Neural Networks). It is difficult to overestimate the importance of understanding the complexity of the Fourier transform.
In the beginning of the project, the PI has developed a mathematical, extremely elegant technique for tackling this problem: The quasi-entropy method. Using this technique, he was able to show a striking tradeoff between the speed of computing the Fourier transform and loss of information due to loss of numerical accuracy. This method has been used to obtain initial results en route the main objective described above. It is believed by the PI that extension of these techniques using more advanced mathematics can bring us closer to the main objective. Additionally, there are certain "derivations of the Fourier transform" (algorithms that are based in some way on the Fourier transform) for which deriving computational lower bounds may be easier, and it is hoped that some of the results of this project will address these smaller, less ambitious problems.
The project is important for society, because the Fourier transform is probably the most important linear transformation used in area such as signal processing, audio/image/video compressing and more. It is used in big polynomial and integer multiplication (core building blocks of modern crypto) and also in machine learning, especially for deep networks that are based on the convolution operator (Convolutional Neural Networks). It is difficult to overestimate the importance of understanding the complexity of the Fourier transform.
In the beginning of the project, the PI has developed a mathematical, extremely elegant technique for tackling this problem: The quasi-entropy method. Using this technique, he was able to show a striking tradeoff between the speed of computing the Fourier transform and loss of information due to loss of numerical accuracy. This method has been used to obtain initial results en route the main objective described above. It is believed by the PI that extension of these techniques using more advanced mathematics can bring us closer to the main objective. Additionally, there are certain "derivations of the Fourier transform" (algorithms that are based in some way on the Fourier transform) for which deriving computational lower bounds may be easier, and it is hoped that some of the results of this project will address these smaller, less ambitious problems.
The first work is a type of "derivation from the Fourier Transform problem". It is documented in a manuscript:
“The Complexity of Computing a Fourier Perturbation” (Together with Gal Yehuda). This paper studies the problem of computing a perturbation of the Fourier transform, defined as the identity matrix plus epsilon (a small perturbation number) plus the Fourier transform matrix. Previous work by the PI showed that a lower bound of epsilon^2 n log n exists for computing the perturbation transformation, using the quasi-entropy method (invented by the PI). However, the result in the paper implies the stronger lower bound of n log n / log(1/epsilon). This seems optimal, because computing with accuracy epsilon requires log(1/epsilon) bits, which cancels out the log(1/epsilon) speedup in the running time. Unfortunately the paper’s result works only for the case that F is a symmetric transform Fourier. For the case that F is asymmetric, the result does not generalize. An important open problem which the PI is working on now is how to generalize the result to the asymmetric case.
In the second work, the PI looked closely at the notion of "condition number" that is often assumed in literature, which is a measure of numerical accuracy of computation. He identifies a more reasonable notion, called "algebraic condition number" that lends itself to improved lower bounds. This is described int the manuscript: “Paraunitary Matrices, Entropy, Algebraic Condition Number and Fourier Computation”: This work has appeared in Journal TCS (Theoretical Computer Science) pending minor revisions, as of June 2019.
Additionally, the following results have been obtained:
- In work under submission, it a tradeoff has been show between dimensionality reduction and NTK (Neural Tangent Kernel) guarantees.
- Various papers on the subject of information theory in the context of graph properties have been accepted to conferences UAI, FCT, WALCOM and AAAI
- Various papers on the subject of information theory in communication have been accepted to ISIT conference.
- A paper in the subject of information tradeoff between accuracy and speed of deep networks has been accepted to conference UAI.
- Various papers in the topic of tradeoffs between side information and the computational ability to take advantage of such information in the context of cooperating agents has been accepted to conferences AAMAS, AISTATS and ICML.
“The Complexity of Computing a Fourier Perturbation” (Together with Gal Yehuda). This paper studies the problem of computing a perturbation of the Fourier transform, defined as the identity matrix plus epsilon (a small perturbation number) plus the Fourier transform matrix. Previous work by the PI showed that a lower bound of epsilon^2 n log n exists for computing the perturbation transformation, using the quasi-entropy method (invented by the PI). However, the result in the paper implies the stronger lower bound of n log n / log(1/epsilon). This seems optimal, because computing with accuracy epsilon requires log(1/epsilon) bits, which cancels out the log(1/epsilon) speedup in the running time. Unfortunately the paper’s result works only for the case that F is a symmetric transform Fourier. For the case that F is asymmetric, the result does not generalize. An important open problem which the PI is working on now is how to generalize the result to the asymmetric case.
In the second work, the PI looked closely at the notion of "condition number" that is often assumed in literature, which is a measure of numerical accuracy of computation. He identifies a more reasonable notion, called "algebraic condition number" that lends itself to improved lower bounds. This is described int the manuscript: “Paraunitary Matrices, Entropy, Algebraic Condition Number and Fourier Computation”: This work has appeared in Journal TCS (Theoretical Computer Science) pending minor revisions, as of June 2019.
Additionally, the following results have been obtained:
- In work under submission, it a tradeoff has been show between dimensionality reduction and NTK (Neural Tangent Kernel) guarantees.
- Various papers on the subject of information theory in the context of graph properties have been accepted to conferences UAI, FCT, WALCOM and AAAI
- Various papers on the subject of information theory in communication have been accepted to ISIT conference.
- A paper in the subject of information tradeoff between accuracy and speed of deep networks has been accepted to conference UAI.
- Various papers in the topic of tradeoffs between side information and the computational ability to take advantage of such information in the context of cooperating agents has been accepted to conferences AAMAS, AISTATS and ICML.
The manuscripts mentioned in the previous box are undoubtedly the state of the art in our understanding of the complexity of Fourier Transform computation.
Toward the end of the project, I believe that the problem of the "asymmetric case" (mentioned in the first paragraph of the previous box) will be settled using tools from group representation theory. Assuming this is achieved, I believe that it will also be possible to obtain a very strong lower bound for the Fourier transform computation in the framework of "algebraic condition number" defined in the paper mention in the previous box, because the two problems are somewhat related.
As is usually the case with theoretical research, it is difficult to anticipate more results. However, there is good chance that other computational lower bound problems defined as "derivations of Fourier Transform" (such as: fast dimensionality reduction, which uses the Fourier transform as a core building block) will be better understood.
Toward the end of the project, I believe that the problem of the "asymmetric case" (mentioned in the first paragraph of the previous box) will be settled using tools from group representation theory. Assuming this is achieved, I believe that it will also be possible to obtain a very strong lower bound for the Fourier transform computation in the framework of "algebraic condition number" defined in the paper mention in the previous box, because the two problems are somewhat related.
As is usually the case with theoretical research, it is difficult to anticipate more results. However, there is good chance that other computational lower bound problems defined as "derivations of Fourier Transform" (such as: fast dimensionality reduction, which uses the Fourier transform as a core building block) will be better understood.