Periodic Reporting for period 3 - E-DUALITY (Exploring Duality for Future Data-driven Modelling)
Période du rapport: 2021-10-01 au 2023-03-31
Future data-driven modelling is increasingly challenging for many systems due to higher complexity levels, such as in energy systems, environmental and climate modelling, traffic and transport, industrial processes, health, safety, and others. It is desirable to conceive new frameworks for tailoring models to the systems and data characteristics, related to tasks of regression, classification, clustering, dimensionality reduction, outlier detection and dynamical systems modelling. An important element at this point is to have a good understanding of representations of models.
The aim of the E-DUALITY project is to explore and engineer the potential of duality principles for future data-driven modelling. Duality principles in general play an important role in mathematics, physics, optimization. Within the context of this project it enables to study different representations of models. For example in support vector machines, models can be represented in primal and dual forms by feature maps or kernel functions, respectively. Depending on the dimensionality of the input space and the number of training data, one representation can be more suitable to employ than another. Another recent example is conjugate feature duality in restricted kernel machines which enables to establish new unexpected connections between kernel machines, neural networks and deep learning.
The overall objective of the project is to obtain a generically applicable framework with unifying insights that includes both parametric and kernel-based approaches, and is applicable to problems with different system complexity levels.
The aim of the E-DUALITY project is to explore and engineer the potential of duality principles for future data-driven modelling. Duality principles in general play an important role in mathematics, physics, optimization. Within the context of this project it enables to study different representations of models. For example in support vector machines, models can be represented in primal and dual forms by feature maps or kernel functions, respectively. Depending on the dimensionality of the input space and the number of training data, one representation can be more suitable to employ than another. Another recent example is conjugate feature duality in restricted kernel machines which enables to establish new unexpected connections between kernel machines, neural networks and deep learning.
The overall objective of the project is to obtain a generically applicable framework with unifying insights that includes both parametric and kernel-based approaches, and is applicable to problems with different system complexity levels.
We have worked on achieving a unifying framework between deep learning, neural networks and kernel machines. Most promising up till now are Restricted Kernel Machine representations of models. It enables to work either in a parametric way (e.g. deep neural networks, convolutional feature maps) or kernel-based in its dual representation. Because of its connection with Restricted Boltzmann machines, it also provides a setting for generative modelling. Moreover, it is suitable for multi-view and tensor based models, deep learning, latent space exploration, explainability and robustness.
For handling different loss functions and achieving robustness, weighted conjugate feature duality is proposed. Deep restricted kernel machines for unsupervised learning are studied by imposing additional orthogonality constraints. Related to conjugate feature duality and Restricted Kernel Machines, a new approach to out-of-distribution detection was proposed related to Stiefel-Restricted Kernel Machines. The Stiefel-Restricted Kernel Machine model is based on manifold optimization techniques, for which a method for disentangled representation learning and generation has been proposed. The proposed method is also suitable for handing large scale problems.
Related to scalability aspects, new insights have been achieved on diversity sampling as an implicit regularization for kernel methods. It combines aspects of determinantal point processes, kernel methods, regularization, sampling and large scale methods. It also leads to new approaches to avoid mode collapse in GANs. For kernel regression in high dimensions refined analysis beyond double descent has been achieved. This is also relevant for a better understanding of the generalization properties of overparameterized neural networks. Related to optimal transport Wasserstein exponential kernels have been proposed.
Recent publications
Fanuel M., Schreurs J., Suykens J.A.K. ``Diversity sampling is an implicit regularization for kernel methods'', SIAM Journal on Mathematics of Data Science (SIMODS), vol. 3, no. 1, Feb. 2021, pp. 280-297.
Liu F., Liao Z., Suykens J.A.K. Kernel regression in high dimensions: Refined analysis beyond double descent, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 130:649-657, 2021
Pandey A., Schreurs J., Suykens J.A.K. Generative Restricted Kernel Machines: A Framework for Multi-view Generation and Disentangled Feature Learning, Neural Networks, Vol.135 pp 177-191, March 2021
Pandey A., Schreurs J., Suykens J.A.K. ``Robust Generative Restricted Kernel Machines using Weighted Conjugate Feature Duality'', LOD 2020, Siena, Italy, pp. 613-624.
For handling different loss functions and achieving robustness, weighted conjugate feature duality is proposed. Deep restricted kernel machines for unsupervised learning are studied by imposing additional orthogonality constraints. Related to conjugate feature duality and Restricted Kernel Machines, a new approach to out-of-distribution detection was proposed related to Stiefel-Restricted Kernel Machines. The Stiefel-Restricted Kernel Machine model is based on manifold optimization techniques, for which a method for disentangled representation learning and generation has been proposed. The proposed method is also suitable for handing large scale problems.
Related to scalability aspects, new insights have been achieved on diversity sampling as an implicit regularization for kernel methods. It combines aspects of determinantal point processes, kernel methods, regularization, sampling and large scale methods. It also leads to new approaches to avoid mode collapse in GANs. For kernel regression in high dimensions refined analysis beyond double descent has been achieved. This is also relevant for a better understanding of the generalization properties of overparameterized neural networks. Related to optimal transport Wasserstein exponential kernels have been proposed.
Recent publications
Fanuel M., Schreurs J., Suykens J.A.K. ``Diversity sampling is an implicit regularization for kernel methods'', SIAM Journal on Mathematics of Data Science (SIMODS), vol. 3, no. 1, Feb. 2021, pp. 280-297.
Liu F., Liao Z., Suykens J.A.K. Kernel regression in high dimensions: Refined analysis beyond double descent, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 130:649-657, 2021
Pandey A., Schreurs J., Suykens J.A.K. Generative Restricted Kernel Machines: A Framework for Multi-view Generation and Disentangled Feature Learning, Neural Networks, Vol.135 pp 177-191, March 2021
Pandey A., Schreurs J., Suykens J.A.K. ``Robust Generative Restricted Kernel Machines using Weighted Conjugate Feature Duality'', LOD 2020, Siena, Italy, pp. 613-624.
In the current state of the art kernel-based models such as support vector machines are often opposed to neural networks and deep learning. However, thanks to the new insights that we obtained on new synergies and a unifying framework between kernel machines, neural networks and deep learning, the results of this project are going beyond this state of the art. The duality principles play a key role in achieving this. New unexpected connections have been found through conjugate feature duality between kernel principal component analysis, least squares support vector machines, restricted Boltzmann machines and deep Boltzmann machines.
Further expected results are in the direction of dynamical systems modelling, manifold learning and networks, adversarial robustness, exploring synergies between duality principles in GANs, RKMs and optimal transport, and the search for new standard forms and template algorithms towards systems with different complexity levels.
Further expected results are in the direction of dynamical systems modelling, manifold learning and networks, adversarial robustness, exploring synergies between duality principles in GANs, RKMs and optimal transport, and the search for new standard forms and template algorithms towards systems with different complexity levels.