Periodic Reporting for period 4 - SOLARIS (Large-Scale Learning with Deep Kernel Machines)
Período documentado: 2021-09-01 hasta 2022-02-28
Machine learning is a recent scientific field at the crossing of statistics,
computer science, and applied mathematics. It may be seen as the science of
data analysis and potentially impacts every domain where data has become a
potential source of knowledge or economic activity. It has become a key part
of scientific fields that produce a massive amount of data and that are in dire
need of scalable tools to automatically make sense of it. Unfortunately,
classical statistical modeling has often become impractical due to recent
shifts in the amount of data to process, and in the high complexity and large
size of models that are able to take advantage of massive data. The promise of
SOLARIS is to invent a new generation of machine learning models that fulfill
the current needs of large-scale data analysis: high scalability, ability to
deal with huge-dimensional models, fast learning, easiness of use, and
adaptivity to various data structures.
These problems are important for society. Besides a potential impact across
different disciplines, big data is shifting classical paradigms and developing
scalable technology to make sense of massive data has become a strategic issue
for society.
The project has four scientific objectives: (i) Providing more
scalability to nonlinear models in machine learning, and developing learning
schemes that can deal with large model sizes and huge amounts of training data.
(ii) Gaining theoretical insight and principled methodology for deep learning
by marrying two schools of thought that have been considered so far to have
little overlap: kernel methods and deep learning. The former is associated
with a well-understood theory and methodology but lacks scalability, whereas
the latter has obtained significant success on large-scale prediction problems,
notably in computer vision. (iii) Building scalable machine learning techniques
for structured data such as sequences or graphs. (iv) Pushing the frontiers of
image and video modeling.
By focusing on the previous four objectives, the project has led to many
contributions, which range from theory (e.g. for better understanding of deep
neural networks), to algorithms and methods (in machine learning, computer
vision, optimization), software developments in open-source toolboxes, and
real-world applications in various scientific fields (with a strong emphasis on
visual modeling).
computer science, and applied mathematics. It may be seen as the science of
data analysis and potentially impacts every domain where data has become a
potential source of knowledge or economic activity. It has become a key part
of scientific fields that produce a massive amount of data and that are in dire
need of scalable tools to automatically make sense of it. Unfortunately,
classical statistical modeling has often become impractical due to recent
shifts in the amount of data to process, and in the high complexity and large
size of models that are able to take advantage of massive data. The promise of
SOLARIS is to invent a new generation of machine learning models that fulfill
the current needs of large-scale data analysis: high scalability, ability to
deal with huge-dimensional models, fast learning, easiness of use, and
adaptivity to various data structures.
These problems are important for society. Besides a potential impact across
different disciplines, big data is shifting classical paradigms and developing
scalable technology to make sense of massive data has become a strategic issue
for society.
The project has four scientific objectives: (i) Providing more
scalability to nonlinear models in machine learning, and developing learning
schemes that can deal with large model sizes and huge amounts of training data.
(ii) Gaining theoretical insight and principled methodology for deep learning
by marrying two schools of thought that have been considered so far to have
little overlap: kernel methods and deep learning. The former is associated
with a well-understood theory and methodology but lacks scalability, whereas
the latter has obtained significant success on large-scale prediction problems,
notably in computer vision. (iii) Building scalable machine learning techniques
for structured data such as sequences or graphs. (iv) Pushing the frontiers of
image and video modeling.
By focusing on the previous four objectives, the project has led to many
contributions, which range from theory (e.g. for better understanding of deep
neural networks), to algorithms and methods (in machine learning, computer
vision, optimization), software developments in open-source toolboxes, and
real-world applications in various scientific fields (with a strong emphasis on
visual modeling).
The project has led so far to 38 publications in top-tier journals and
conferences in machine learning, computer vision, signal processing,
optimization. Most of these publications have been released with open-source
software packages. These were either simple packages allowing to reproduce the
experiments of our papers, or more ambitious toolboxes with a long-term
development plan. Results obtained in image processing during the last year of
the project have also a strong industrial potential and have led to a start-up
creation project, which is currently under discussion with Inria's technology
transfer department.
Our effort has been divided into four lines of research:
1) Optimization for large-scale machine learning
We have developed optimization techniques that can be applied to a large class
of machine learning models, reducing the training time while offering strong
theoretical guarantees.
2) Theoretical and methodological foundations of deep learning
We have characterized precisely the invariance and stability properties of deep
convolutional neural networks to image perturbations such as deformations, and
have proposed a new framework that relates these properties to the generalization
capability of these networks. Based on these theoretical tools, we have derived
new machine learning models with better regularization properties and better
robustness to adversarial perturbations than classical deep learning models.
3) Machine learning models for structured data
Through pluri-disciplinary collaborations, we have developed predictive
models for biological sequences such as DNA/RNA or proteins, achieving significant
gains compared to deep learning models. Recently, we have introduced several
approaches for representing graph-structured data or data represented as sets
of features, achieving state-of-the-art results on various prediction tasks.
These lines of research have fostered pluri-discipinary collaborations that
are on-going.
4) Visual modeling
We have pushed the state of the art in computer vision in several directions,
For visual recognition, new breakthroughs have been obtained for learning good
visual representations without relying on massive annotations.
We have also introduced several approaches for image restoration achieving
state-of-the-art results for single-image denoising and super-resolution, and
for processing bursts of raw images acquired by smartphones or digital cameras.
conferences in machine learning, computer vision, signal processing,
optimization. Most of these publications have been released with open-source
software packages. These were either simple packages allowing to reproduce the
experiments of our papers, or more ambitious toolboxes with a long-term
development plan. Results obtained in image processing during the last year of
the project have also a strong industrial potential and have led to a start-up
creation project, which is currently under discussion with Inria's technology
transfer department.
Our effort has been divided into four lines of research:
1) Optimization for large-scale machine learning
We have developed optimization techniques that can be applied to a large class
of machine learning models, reducing the training time while offering strong
theoretical guarantees.
2) Theoretical and methodological foundations of deep learning
We have characterized precisely the invariance and stability properties of deep
convolutional neural networks to image perturbations such as deformations, and
have proposed a new framework that relates these properties to the generalization
capability of these networks. Based on these theoretical tools, we have derived
new machine learning models with better regularization properties and better
robustness to adversarial perturbations than classical deep learning models.
3) Machine learning models for structured data
Through pluri-disciplinary collaborations, we have developed predictive
models for biological sequences such as DNA/RNA or proteins, achieving significant
gains compared to deep learning models. Recently, we have introduced several
approaches for representing graph-structured data or data represented as sets
of features, achieving state-of-the-art results on various prediction tasks.
These lines of research have fostered pluri-discipinary collaborations that
are on-going.
4) Visual modeling
We have pushed the state of the art in computer vision in several directions,
For visual recognition, new breakthroughs have been obtained for learning good
visual representations without relying on massive annotations.
We have also introduced several approaches for image restoration achieving
state-of-the-art results for single-image denoising and super-resolution, and
for processing bursts of raw images acquired by smartphones or digital cameras.
There is still a long way to go before being able to build intelligent systems
that are able to learn without massive amounts of data. The SOLARIS project
did nevertheless significant progress along several lines. For instance,
we may consider the following five achievements to be the most important of the
project:
1) Our theoretical understanding of why deep neural networks are successful
have improved, and many recent works (including ours) are adopting the point of
view of kernel methods promoted and developed within the project. In
particular, we have shown that design choices in neural network architectures
(in particular in convolutional neural networks) have a strong impact on the
stability, approximation, and generalization properties of these models.
2) A breakthrough in terms of optimization has been obtained for accelerating
algorithms when minimizing cost functions that often ocur in machine learning.
Algorithms developed within SOLARIS have been implemented in an open-source
software packages offering state-of-the-art performance for learning large-scale
linear models.
3) On a more applied side, our line of research has opened many perspectives
to process structured data such as sequences and graphs. In particular, our
work on graph representations is one of the first trainable architecture that
is more expressive than traditional graph neural networks. Several on-going
pluri-disciplinary collaborations in computational biology are based on this
outcome.
4) Major achievements have been obtained in computer vision for unsupervised
learning of visual representations, where we have shown that massive annotations
are not required to obtain representations that perform well on many tasks.
5) In image processing, we have developed approaches that significantly push the
state of the art for solving inverse problems. This will be the beginning of a line
of work that offers many perspectives for both scientific and technological
applications. This will be pursued after the end of the SOLARIS project.
This first results are currently the focus of a start-up creation project.
that are able to learn without massive amounts of data. The SOLARIS project
did nevertheless significant progress along several lines. For instance,
we may consider the following five achievements to be the most important of the
project:
1) Our theoretical understanding of why deep neural networks are successful
have improved, and many recent works (including ours) are adopting the point of
view of kernel methods promoted and developed within the project. In
particular, we have shown that design choices in neural network architectures
(in particular in convolutional neural networks) have a strong impact on the
stability, approximation, and generalization properties of these models.
2) A breakthrough in terms of optimization has been obtained for accelerating
algorithms when minimizing cost functions that often ocur in machine learning.
Algorithms developed within SOLARIS have been implemented in an open-source
software packages offering state-of-the-art performance for learning large-scale
linear models.
3) On a more applied side, our line of research has opened many perspectives
to process structured data such as sequences and graphs. In particular, our
work on graph representations is one of the first trainable architecture that
is more expressive than traditional graph neural networks. Several on-going
pluri-disciplinary collaborations in computational biology are based on this
outcome.
4) Major achievements have been obtained in computer vision for unsupervised
learning of visual representations, where we have shown that massive annotations
are not required to obtain representations that perform well on many tasks.
5) In image processing, we have developed approaches that significantly push the
state of the art for solving inverse problems. This will be the beginning of a line
of work that offers many perspectives for both scientific and technological
applications. This will be pursued after the end of the SOLARIS project.
This first results are currently the focus of a start-up creation project.