Periodic Reporting for period 3 - SPECGEO (Spectral geometric methods in practice)
Okres sprawozdawczy: 2021-09-01 do 2023-02-28
Geometric data (broadly understood as shape, connectivity or proximity) are everywhere: Our body, its parts, and the world around us are three-dimensional; proteins have a complex three-dimensional structure, carrying essential information about their biological function; the functional and anatomical connectivity of the brain make up the human connectome, a multi-scale representation encompassing different modalities; social networks model users and their relationships as signals on a graph; deep learning architectures have a natural topological representation as networks; collections of sensors, images, shapes, operators and camera motions occur in multiple key tasks in computer vision, where they are represented via attributed graphs or dynamic networks. On the one hand, the ever-growing role of computers and digital sensors has impacted virtually every aspect of our life, leading to an exponential growth in the amount and complexity of the data we collect. On the other hand, in this new epoch of “big data”, we can no longer rely on careful supervision (i.e. a person looking at the data). We must be able to cope with all kinds of artifacts impeding progress at all phases of the pipeline that creates knowledge from geometry. Missing and corrupted data is the norm and not the exception in any real-world setting, and we need to ensure that the observations we gather are still useful and enlightening, and develop the supporting methodologies to use digital geometric data with confidence.
Today, the application of spectral techniques to (possibly abstract and high-dimensional) geometric data undergoing real-world transformations such as deformation, incompleteness, topological noise and other forms of corruption is severely limited. These problems pose new challenges that cannot be attacked by a direct adaptation of existing machinery: A paradigm shift is needed in order to develop new methods that are robust to grossly corrupted input, at the same time offering strong guarantees on the output. This project aims at developing a new generation of spectral methods for the analysis and processing of geometric data undergoing complex transformations, strong partiality, topological alterations and other forms of massive corruption that are typically observed in real-world scenarios. We believe that there is a timely opportunity for such an inquiry, a view that is apparently shared by prominent colleagues across different communities. The amount of new challenges arising in these fields are enormous, and their solution requires fundamentally new ideas and approaches, with significant consequences in practical applications.
The ultimate goal of this project is to devise a theoretical spectral model and computational framework for analyzing and processing geometric data undergoing complex transformations, including topology changes, partiality and noise, as well as to rigorously address aspects related to the data representation, such as cross-modality and high dimensionality. We will focus our research toward applied areas where geometric data play a crucial role, such as computer vision, graphics, and machine learning, with further applications to computational biology. As the scope of spectral data analysis and geometry goes far beyond these areas, this project will have major impact in other sub-fields of natural sciences and engineering, such as acoustics, mechanical engineering, and computational sociology.
Today, the application of spectral techniques to (possibly abstract and high-dimensional) geometric data undergoing real-world transformations such as deformation, incompleteness, topological noise and other forms of corruption is severely limited. These problems pose new challenges that cannot be attacked by a direct adaptation of existing machinery: A paradigm shift is needed in order to develop new methods that are robust to grossly corrupted input, at the same time offering strong guarantees on the output. This project aims at developing a new generation of spectral methods for the analysis and processing of geometric data undergoing complex transformations, strong partiality, topological alterations and other forms of massive corruption that are typically observed in real-world scenarios. We believe that there is a timely opportunity for such an inquiry, a view that is apparently shared by prominent colleagues across different communities. The amount of new challenges arising in these fields are enormous, and their solution requires fundamentally new ideas and approaches, with significant consequences in practical applications.
The ultimate goal of this project is to devise a theoretical spectral model and computational framework for analyzing and processing geometric data undergoing complex transformations, including topology changes, partiality and noise, as well as to rigorously address aspects related to the data representation, such as cross-modality and high dimensionality. We will focus our research toward applied areas where geometric data play a crucial role, such as computer vision, graphics, and machine learning, with further applications to computational biology. As the scope of spectral data analysis and geometry goes far beyond these areas, this project will have major impact in other sub-fields of natural sciences and engineering, such as acoustics, mechanical engineering, and computational sociology.
On the theoretical front, eigenspace dynamics have been addressed with success and published within the international community of digital geometry processing; this has been a first important step toward understanding the change of eigenfunctions due to geometric perturbation (e.g. missing data), which has been previously observed but never investigated thoroughly. Inverse spectral geometric problems have also been studied in depth, and are the main focus of several of our publications in the last several months. Our work in this direction has received prizes; we went beyond what we originally envisioned in the proposal, and were able to devise both axiomatic (learning-free) and data-driven (learning-based) computational approaches to attack the inverse spectral geometric problem ("hearing the shape of the drum"). Finally, maps and correspondence problems are featured in most of our recent publications as a sort of application "playground", since they lend themselves particularly well to showcase our computational techniques. A more particular case concerns the sub-goal of exploring the spectral representation of maps on top of product manifolds, which was explored in great detail and published as the project started.
Perhaps the most unconventional among our methodologies are the shape-from-spectrum techniques we developed (and are still developing) to tackle the inverse problems. While these were originally one of many sub-goals of the project, the published techniques were very welcome by the community, and further developments are now undergoing. One surprising development lies in the discovery that inverse problems can be used to formulate spectral adversarial attacks on geometric data such as deformable 3D shapes. This exciting discovery is at the basis of further research we are conducting in this direction.
The computational framework conceived in the proposal has been under development since the very first months, and our software library is shaping up in parallel with the theoretical advances we make on the project. In particular, we are adopting an incremental approach, according to which we enrich the library with new functionalities for each new publication. At the current stage, the library is not a monolithic piece of software; rather, it is composed of several modules, stored in as many repositories (e.g. github); by the end of the project, we expect to have integrated all the modules in one universal framework, which will be ready for adoption by the applied data sciences community at large.
Perhaps the most unconventional among our methodologies are the shape-from-spectrum techniques we developed (and are still developing) to tackle the inverse problems. While these were originally one of many sub-goals of the project, the published techniques were very welcome by the community, and further developments are now undergoing. One surprising development lies in the discovery that inverse problems can be used to formulate spectral adversarial attacks on geometric data such as deformable 3D shapes. This exciting discovery is at the basis of further research we are conducting in this direction.
The computational framework conceived in the proposal has been under development since the very first months, and our software library is shaping up in parallel with the theoretical advances we make on the project. In particular, we are adopting an incremental approach, according to which we enrich the library with new functionalities for each new publication. At the current stage, the library is not a monolithic piece of software; rather, it is composed of several modules, stored in as many repositories (e.g. github); by the end of the project, we expect to have integrated all the modules in one universal framework, which will be ready for adoption by the applied data sciences community at large.
Spectral methods have had a lasting impact in applied sciences, bringing unprecedented performance and novel capabilities to practically all fields dealing with geometric data. By operating at a fundamental level, rather than devising ad-hoc solutions for specific tasks, our methodology will be applicable to a spectrum of open problems arising in many fields, and has the potential to be addressed to still unthinkable problems that are likely to emerge in the years to come. The incredible growth of geometric data that we have been witnessing in many domains of science in the past years makes this a timely opportunity for such a research.
We can already see tangible benefits of our studies. In the realm of computer vision and graphics, where geometric data has been in the bull’s eye since the inception of these fields, we have been able to go beyond the state of the art in shape correspondence problems using our spectral methods, either in a purely axiomatic fashion, or within a data-driven paradigm. Especially relevant is the fact that the proposed approaches allow to address notorious challenges in the field, namely, the presence of geometry artifacts, partiality, or lack of isometry. This has been an important step toward overcoming the general skepticism of the geometry processing community – apparently discouraged by the claimed instability of spectral constructions transpiring from classical theory.
Perhaps less expected has been the resounding success we encountered while tackling our sub-goals on inverse spectral geometric problems ("hearing the shape of the drum" problems) and adversarial attacks on geometric data. The former sub-goal has been addressed with three top-level publications so far, two of which have been awarded Best Paper prizes at international conferences; this direction is especially promising, and judging from the impact on the scientific community (in terms of citations and accolades), we anticipate this to become a major and fruitful avenue of investigation for the months to come. The latter sub-goal (spectral adversarial perturbations) has been less problematic than originally anticipated, resulting in two publications at major international venues. Part of this success is also due to the advances we made on the shape-of-the-drum problems, whose flexible methodology we were able to exploit as an effective computational tool.
We can already see tangible benefits of our studies. In the realm of computer vision and graphics, where geometric data has been in the bull’s eye since the inception of these fields, we have been able to go beyond the state of the art in shape correspondence problems using our spectral methods, either in a purely axiomatic fashion, or within a data-driven paradigm. Especially relevant is the fact that the proposed approaches allow to address notorious challenges in the field, namely, the presence of geometry artifacts, partiality, or lack of isometry. This has been an important step toward overcoming the general skepticism of the geometry processing community – apparently discouraged by the claimed instability of spectral constructions transpiring from classical theory.
Perhaps less expected has been the resounding success we encountered while tackling our sub-goals on inverse spectral geometric problems ("hearing the shape of the drum" problems) and adversarial attacks on geometric data. The former sub-goal has been addressed with three top-level publications so far, two of which have been awarded Best Paper prizes at international conferences; this direction is especially promising, and judging from the impact on the scientific community (in terms of citations and accolades), we anticipate this to become a major and fruitful avenue of investigation for the months to come. The latter sub-goal (spectral adversarial perturbations) has been less problematic than originally anticipated, resulting in two publications at major international venues. Part of this success is also due to the advances we made on the shape-of-the-drum problems, whose flexible methodology we were able to exploit as an effective computational tool.