Final Report Summary - CHESS (Challenges in Extraction and Separation of Sources)
The CHESS project addresses 3 challenges in extraction and separation of sources.
The first challenge concerns source separation for multimodal recordings. In fact, multimodal recordings can be due to different devices (e.g. EEG and MEG in brain imaging), different time (space) windows for studying dynamics of data along time (space), or different subjects (e.g. patients) recorded by the same device. Although these situations are very different, from a theoretical point of view, they require to jointly process multiple datasets (one per modality) with interactions between them. This challenge relates to data fusion, but the main goal in CHESS is to develop comprehensive foundations and generic methods for multimodal processing instead of designing ad hoc algorithms.
We propose a new source separation model assuming multidimensional sources and multimodal recordings. This model extends independent component analysis (ICA), independent vector analysis (IVA) and independent subspace analysis (ISA). Results, some of them based on generalization of Schur’s Lemma, show that multimodality, provided that hard or soft interactions exist between datasets, leads to relaxed conditions for source identifiability and uniqueness.
The core of multimodal models is the interaction between datasets. For multi-devices or
multi-temporal data sets, we develop a general and flexible framework suited to a vast class of
models with interaction, e.g. when datasets share common, correlated or weakly related factors, or with factors varying along data sets. This leads to algorithmic implementations, based on non convex optimization with constraints: the cost function contains a classical data fit term, completed by regularization terms, modeling interactions between the datasets.
More generally, performance in joint processing of multimodal recordings is usually assumed better than that achieved using only one recording coming from a unique modality. But, in the literature, there are results in contradiction with this claim. We then studied, in an information theoretic approach, the benefits or disadvantages of using two or more modalities. Our results explain how different sampling rates, SNRs in each modality and correlation between modalities influence estimation performance.
The second challenge focuses on source separation in nonlinear mixtures. A new generic approach consists in replacing the time-invariant nonlinear mixture of sources by a time-varying linear mixture of the derivatives of the sources. This idea only requires mild conditions, i.e. the nonlinear model to be differentiable and sources to be smooth enough and with statistically independent derivatives. It leads to theoretical proof of identifiability and new algorithms. A second, very generic approach too, is based on the fact that the Gaussian process property is lost when mixed nonlinearly with polynomial. Thus, Gaussian process can be used as a criterion for separating colored sources satisfying Gaussian process model, using simple second-order statistics. Main applications are focused on processing signals coming from ion-sensitive or gas sensor arrays.
The third challenge (extraction of sources in high- or low-dimension data) has been explored in three multimodal applicative frameworks: PCG/ECG based non-invasive fetal heart extraction, audio-video speech separation, gaze-EEG recordings, and hyperscanning. Typically, we design methods, which exploit simple hints of the sources of interest: hints can be properties like quasi-periodicity or simple binary information coming from one modality.
For hyperscanning, we show that approximate joint diagonalizer of a set of matrices is related to the geometric mean of those matrices. This finding links blind source separation to classification on Riemannian manifold.
All the CHESS publications, can be found on the open-access site HAL: https://hal.archives-ouvertes.fr/ using the acronym CHESS in “European project” topics.
The first challenge concerns source separation for multimodal recordings. In fact, multimodal recordings can be due to different devices (e.g. EEG and MEG in brain imaging), different time (space) windows for studying dynamics of data along time (space), or different subjects (e.g. patients) recorded by the same device. Although these situations are very different, from a theoretical point of view, they require to jointly process multiple datasets (one per modality) with interactions between them. This challenge relates to data fusion, but the main goal in CHESS is to develop comprehensive foundations and generic methods for multimodal processing instead of designing ad hoc algorithms.
We propose a new source separation model assuming multidimensional sources and multimodal recordings. This model extends independent component analysis (ICA), independent vector analysis (IVA) and independent subspace analysis (ISA). Results, some of them based on generalization of Schur’s Lemma, show that multimodality, provided that hard or soft interactions exist between datasets, leads to relaxed conditions for source identifiability and uniqueness.
The core of multimodal models is the interaction between datasets. For multi-devices or
multi-temporal data sets, we develop a general and flexible framework suited to a vast class of
models with interaction, e.g. when datasets share common, correlated or weakly related factors, or with factors varying along data sets. This leads to algorithmic implementations, based on non convex optimization with constraints: the cost function contains a classical data fit term, completed by regularization terms, modeling interactions between the datasets.
More generally, performance in joint processing of multimodal recordings is usually assumed better than that achieved using only one recording coming from a unique modality. But, in the literature, there are results in contradiction with this claim. We then studied, in an information theoretic approach, the benefits or disadvantages of using two or more modalities. Our results explain how different sampling rates, SNRs in each modality and correlation between modalities influence estimation performance.
The second challenge focuses on source separation in nonlinear mixtures. A new generic approach consists in replacing the time-invariant nonlinear mixture of sources by a time-varying linear mixture of the derivatives of the sources. This idea only requires mild conditions, i.e. the nonlinear model to be differentiable and sources to be smooth enough and with statistically independent derivatives. It leads to theoretical proof of identifiability and new algorithms. A second, very generic approach too, is based on the fact that the Gaussian process property is lost when mixed nonlinearly with polynomial. Thus, Gaussian process can be used as a criterion for separating colored sources satisfying Gaussian process model, using simple second-order statistics. Main applications are focused on processing signals coming from ion-sensitive or gas sensor arrays.
The third challenge (extraction of sources in high- or low-dimension data) has been explored in three multimodal applicative frameworks: PCG/ECG based non-invasive fetal heart extraction, audio-video speech separation, gaze-EEG recordings, and hyperscanning. Typically, we design methods, which exploit simple hints of the sources of interest: hints can be properties like quasi-periodicity or simple binary information coming from one modality.
For hyperscanning, we show that approximate joint diagonalizer of a set of matrices is related to the geometric mean of those matrices. This finding links blind source separation to classification on Riemannian manifold.
All the CHESS publications, can be found on the open-access site HAL: https://hal.archives-ouvertes.fr/ using the acronym CHESS in “European project” topics.