Periodic Reporting for period 3 - E-DUALITY (Exploring Duality for Future Data-driven Modelling)
Berichtszeitraum: 2021-10-01 bis 2023-03-31
The aim of the E-DUALITY project is to explore and engineer the potential of duality principles for future data-driven modelling. Duality principles in general play an important role in mathematics, physics, optimization. Within the context of this project it enables to study different representations of models. For example in support vector machines, models can be represented in primal and dual forms by feature maps or kernel functions, respectively. Depending on the dimensionality of the input space and the number of training data, one representation can be more suitable to employ than another. Another recent example is conjugate feature duality in restricted kernel machines which enables to establish new unexpected connections between kernel machines, neural networks and deep learning.
The overall objective of the project is to obtain a generically applicable framework with unifying insights that includes both parametric and kernel-based approaches, and is applicable to problems with different system complexity levels.
For handling different loss functions and achieving robustness, weighted conjugate feature duality is proposed. Deep restricted kernel machines for unsupervised learning are studied by imposing additional orthogonality constraints. Related to conjugate feature duality and Restricted Kernel Machines, a new approach to out-of-distribution detection was proposed related to Stiefel-Restricted Kernel Machines. The Stiefel-Restricted Kernel Machine model is based on manifold optimization techniques, for which a method for disentangled representation learning and generation has been proposed. The proposed method is also suitable for handing large scale problems.
Related to scalability aspects, new insights have been achieved on diversity sampling as an implicit regularization for kernel methods. It combines aspects of determinantal point processes, kernel methods, regularization, sampling and large scale methods. It also leads to new approaches to avoid mode collapse in GANs. For kernel regression in high dimensions refined analysis beyond double descent has been achieved. This is also relevant for a better understanding of the generalization properties of overparameterized neural networks. Related to optimal transport Wasserstein exponential kernels have been proposed.
Recent publications
Fanuel M., Schreurs J., Suykens J.A.K. ``Diversity sampling is an implicit regularization for kernel methods'', SIAM Journal on Mathematics of Data Science (SIMODS), vol. 3, no. 1, Feb. 2021, pp. 280-297.
Liu F., Liao Z., Suykens J.A.K. Kernel regression in high dimensions: Refined analysis beyond double descent, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 130:649-657, 2021
Pandey A., Schreurs J., Suykens J.A.K. Generative Restricted Kernel Machines: A Framework for Multi-view Generation and Disentangled Feature Learning, Neural Networks, Vol.135 pp 177-191, March 2021
Pandey A., Schreurs J., Suykens J.A.K. ``Robust Generative Restricted Kernel Machines using Weighted Conjugate Feature Duality'', LOD 2020, Siena, Italy, pp. 613-624.
Further expected results are in the direction of dynamical systems modelling, manifold learning and networks, adversarial robustness, exploring synergies between duality principles in GANs, RKMs and optimal transport, and the search for new standard forms and template algorithms towards systems with different complexity levels.