Periodic Reporting for period 3 - SLING (Efficient algorithms for sustainable machine learning)
Okres sprawozdawczy: 2022-11-01 do 2024-04-30
SLING aims at creating a new generation of resource-efficient algorithms for large-scale machine learning by integrating the latest advancements in optimization and statistics. The core premise of this project is that enhancing efficiency is crucial for scaling the ambitions and practical applications of machine learning. SLING's primary objective is to develop lossless compressed algorithms to make machine learning sustainable and accessible.
The world is undergoing a revolution driven by AI systems that rely on machine learning models trained on massive datasets. However, while the potential for profitable and useful applications of AI seems endless, there are concerning aspects of current AI systems: they are opaque black boxes that are poorly understood, and have unsustainable computational and data requirements. These systems come with significant energy costs which greatly limit their applicability, ease of use, and democratic access to this technology. Developing frugal and efficient systems is not only an exciting scientific challenge, but also an urgent need to ensure the sustainability and accessibility of AI.
SLING goes beyond the traditional boundaries between statistics and computations to establish a new theory of algorithm design. It develops statistical models that incorporate budgeted computations and numerical solutions tailored to the statistical accuracy allowed by the available data. To achieve this, SLING follows a research plan with four main objectives. Two objectives focus on reducing memory and time footprint while preserving learning accuracy. Another objective aims at exploiting the complex structure of data, such as its geometry and time-evolving nature. Lastly, there is an applied effort to test the practical relevance of the derived solutions in two diverse contexts: robotics and high-energy physics.
The world is undergoing a revolution driven by AI systems that rely on machine learning models trained on massive datasets. However, while the potential for profitable and useful applications of AI seems endless, there are concerning aspects of current AI systems: they are opaque black boxes that are poorly understood, and have unsustainable computational and data requirements. These systems come with significant energy costs which greatly limit their applicability, ease of use, and democratic access to this technology. Developing frugal and efficient systems is not only an exciting scientific challenge, but also an urgent need to ensure the sustainability and accessibility of AI.
SLING goes beyond the traditional boundaries between statistics and computations to establish a new theory of algorithm design. It develops statistical models that incorporate budgeted computations and numerical solutions tailored to the statistical accuracy allowed by the available data. To achieve this, SLING follows a research plan with four main objectives. Two objectives focus on reducing memory and time footprint while preserving learning accuracy. Another objective aims at exploiting the complex structure of data, such as its geometry and time-evolving nature. Lastly, there is an applied effort to test the practical relevance of the derived solutions in two diverse contexts: robotics and high-energy physics.
The research work flow was built around the four objectives of SLING, namely reducing 1) memory and 2) time footprints, 3) exploiting data structure and 4) test compressed algorithms in real scenarios.
We are designing compressed algorithms by reducing data dimensionality using random and structured projections that can be efficiently computed. We derive computational techniques that can seamlessly take advantage of the structure in the data while being robust to stochastic or deterministic perturbation that might affect them. We do so by combining ideas from inexact optimization with regularization theory for machine learning. Beyond classic supervised learning, we are considering a number of unsupervised and structured learning problems, including the manipulation of probability distribution via embeddings in Hilbert spaces and their applications. We are deriving a wide range of algorithmic solutions accompanied by rigorous theoretical guarantees. The theory we develop quantitatively characterizes the trade-offs between accuracy and efficiency, by blending results from numerics and statistics. Our theoretical findings are the foundation of trustworty algorithmic solutions and corresponding implementations. A dedicated effort is devoted to turning algorithmic ideas into open source software libraries. We make sure these novel compressed AI systems are really effective by testing them in a number of practical problems, with a focus on high-energy physics and humanoid robotics, considering perceptual and motor skill learning tasks.
We are designing compressed algorithms by reducing data dimensionality using random and structured projections that can be efficiently computed. We derive computational techniques that can seamlessly take advantage of the structure in the data while being robust to stochastic or deterministic perturbation that might affect them. We do so by combining ideas from inexact optimization with regularization theory for machine learning. Beyond classic supervised learning, we are considering a number of unsupervised and structured learning problems, including the manipulation of probability distribution via embeddings in Hilbert spaces and their applications. We are deriving a wide range of algorithmic solutions accompanied by rigorous theoretical guarantees. The theory we develop quantitatively characterizes the trade-offs between accuracy and efficiency, by blending results from numerics and statistics. Our theoretical findings are the foundation of trustworty algorithmic solutions and corresponding implementations. A dedicated effort is devoted to turning algorithmic ideas into open source software libraries. We make sure these novel compressed AI systems are really effective by testing them in a number of practical problems, with a focus on high-energy physics and humanoid robotics, considering perceptual and motor skill learning tasks.
SLING adopts a systematic approach of combining ideas and findings from neighboring fields to synthesize and generate innovative solutions. In machine learning, there is often a divide et impera approach where statistical and computational aspects are treated separately and by different scientific communities. Instead, SLING takes a holistic approach and fully leverages an interdisciplinary perspective.
For example, while considering memory compression for machine learning, we elaborated a connection between sketching approaches common in computer science and Galerkin methods developed in numerical analysis, studied using tools from approximation and interpolation theory. We showed that regularization theory for inverse problems, specifically regularization by projection, provides a unified framework when combined with concentration of measures results.
Another example is recognizing that iterative computations can be viewed as defining dynamical systems that adaptively navigate the trade-off between accuracy and efficiency, leading to massive reductions in compute time. Inexact optimization techniques, developed to handle approximate numerical computations, need to be adjusted to address irreducible stochastic errors, which can be studied using probabilistic tools. Once again, regularization theory, including its stochastic extensions, offers a natural framework for analyzing and deriving novel compressed algorithms.
SLING establishes a connection between theory and practice by testing the solutions developed in two diverse domains: high-energy physics and robotics. Our results demonstrate various scenarios where the compressed algorithms designed in SLING are accurate and simultaneously yielding substantial improvements in computational efficiency, often reducing compute time from hours to seconds. Notably, they also alleviate the need for cumbersome computational infrastructure, enabling the utilization of basic workstations.
For example, while considering memory compression for machine learning, we elaborated a connection between sketching approaches common in computer science and Galerkin methods developed in numerical analysis, studied using tools from approximation and interpolation theory. We showed that regularization theory for inverse problems, specifically regularization by projection, provides a unified framework when combined with concentration of measures results.
Another example is recognizing that iterative computations can be viewed as defining dynamical systems that adaptively navigate the trade-off between accuracy and efficiency, leading to massive reductions in compute time. Inexact optimization techniques, developed to handle approximate numerical computations, need to be adjusted to address irreducible stochastic errors, which can be studied using probabilistic tools. Once again, regularization theory, including its stochastic extensions, offers a natural framework for analyzing and deriving novel compressed algorithms.
SLING establishes a connection between theory and practice by testing the solutions developed in two diverse domains: high-energy physics and robotics. Our results demonstrate various scenarios where the compressed algorithms designed in SLING are accurate and simultaneously yielding substantial improvements in computational efficiency, often reducing compute time from hours to seconds. Notably, they also alleviate the need for cumbersome computational infrastructure, enabling the utilization of basic workstations.