Multimodal data analysis
One of the long-standing challenges in signal processing and data analysis is the fusion of information acquired by multiple, multimodal sensors. The problem of information fusion has become particularly central in the wake of recent technological advances, which have led to extensive collection and storage of multimodal data. In our research, we address the problem from a manifold learning/geometric analysis standpoint.
In “O. Katz, R. R. Lederman, R. Talmon, Spectral Flow on the Manifold of SPD Matrices for Multimodal Data Processing”, we combine manifold learning with the Riemannian geometry of symmetric and positive-definite (SPD) matrices. In particular, we study the way the spectra of kernels change along geodesic paths on the manifold of SPD matrices. We show that this change enables us, in a purely unsupervised manner, to derive a compact, yet informative, description of the relations between the multimodal measurements. Based on this result, we present new algorithms for extracting the common latent components and for identifying common and measurement-specific components.
In addition, in “O. Yair, F. Dietrich, R. Mulayoff, R. Talmon, and I. G. Kevrekidis, Spectral Discovery of Jointly Smooth Features for Multimodal Data”, we present a spectral method for deriving functions that are jointly smooth on multiple observed manifolds. We demonstrate the proposed method on sleep stage identification, and we show how the proposed method can be leveraged for finding minimal realizations of parameter spaces of nonlinear dynamical systems.
The Riemannian geometry of the manifold of diffusion operators
We study the manifold of diffusion operators, on which we can define geometric, differential, and probabilistic structures. This research direction entails a fresh approach to manifold learning, departing from the traditional use of spectral decomposition of diffusion operators for embedding.
Often, diffusion operators are not strictly positive, but rather positive semi-definite. Unlike the manifold of SPD matrices, which has been studied for many years and have a solid theory, the manifold of symmetric positive semi-definite (SPSD) matrices lacks many key results. In “O. Yair, A. Lahav, and R. Talmon, Symmetric Positive Semi-definite Riemannian Geometry with Application to Domain Adaptation”, we present new results on the Riemannian geometry of SPSD matrices, including approximations of the logarithmic maps, the exponential map, and Parallel Transport (PT), as well as a canonical representation for a set of SPSD matrices. Based on these results, we propose an algorithm for Domain Adaptation (DA) and demonstrate its performance in two applications: fusion of hyper-spectral images and motion recognition.
Applications
In “Lustig, E., Yair, O., Talmon, R. and Segev, M., 2020. Identifying Topological Phase Transitions in Experiments Using Manifold Learning. Physical Review Letters, 125(12), p.127401”, we demonstrate the identification of topological phase transitions from experimental data using diffusion maps: a nonlocal unsupervised machine learning method. We analyze experimental data from an optical system undergoing a topological phase transition and demonstrate the ability of this approach to identify topological phase transitions even when the data originates from a small part of the system, and does not even include edge states.
In “Y.-W. E. Lin, T. Shnitzer, R. Talmon, F. Villarroel-Espindola, S. Desai, K. Schalper, and Y. Kluger, Graph of graphs analysis for multiplexed data with application to imaging mass cytometry”, we propose a two-step graph-based analyses for high-dimensional multiplexed datasets characterizing ROIs and their inter-relationships. We apply our method to imaging mass cytometry (IMC) and show that it leads to state of the art prediction of sensitivity to treatment of lung cancer patients.
