Periodic Reporting for period 4 - LEMAN (Deep LEarning on MANifolds and graphs) Reporting period: 2021-10-01 to 2021-12-31 Summary of the context and overall objectives of the project In the past decade, deep learning, a class of machine learning methods based on artificial neural networks, has seen a dramatic success in a wide range of applications, which has literally shaken the academic and industrial world. The success of deep learning has however been primarily limited to applications such as image recognition and speech analysis dealing with data underlying Euclidean grid-like structure (e.g. acoustic signals, images, and videos). There are however many other fields where the data presents non-Euclidean structure: prominent examples include social graphs in social science, regulatory and interaction networks in biology, 3D meshes in computer vision and graphics, etc. Deep learning methods cannot be straightforwardly applied to such data as the very definition of basic operations used in deep networks is rather elusive on non-Euclidean domains. Project LEMAN (“Deep LEarning on MANifolds and graphs”) was born with the ambitious goal to develop a principled framework for generalizing deep learning paradigms to data on non-Euclidean domains, all the way from a mathematical model to an efficient and scalable software implementation. Our work on this project has resulted in pioneering methods for geometric deep learning on graphs and manifolds and their applications to some of the most important and challenging problems in the domains of computer graphics and vision, biochemistry, and physical sciences. Geometric ML methods have led to a leap in performance on several known tough problems and allowed addressing new and previously unthinkable problems. Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far 1. ScienceOur team has pioneered the field of Geometric ML (including coining the term) and helped shape it into one of the most exciting current topics in ML research from a niche community in a matter of a few years. We have proposed some first and already popular architectures for deep learning on manifolds and graphs achieving state-of-the-art results in classically challenging problems in computer graphics and vision such as deformable dense correspondence. We also explored the use of geometric deep learning for the defence against adversarial attacks, recommender systems, astrophysics (our paper on neutrino detection with Geometric DL in collaboration with Berkeley, NYU, and IceCube won the best paper award at ICMLA), fake news detection (our technology was commercialized and acquired by Twitter), protein science (our paper on geometric DL-based protein analysis appeared on the cover of Nature Methods), and even molecular gastronomy (our paper on using graph methods for finding drug-like molecules in food appeared in Nature Scientific Report and attracted significant media attention; the same algorithmic pipeline is now used for drug repositioning against COVID-19). 2. TechnologyWe patented our geometric deep learning technology (two granted patents and several additional patents pending) and spun it off into a startup company Fabula AI funded by ERC Proof of Concept grant “GoodNews” (received by the PI in 2018) as well as additional grants from the industry (Google, Facebook, and Amazon) and private investment. Our PhD student Federico Monti took the role of CTO in the company. The company was acquired by Twitter in 2019. 3. DisseminationThe increasing popularity of Geometric ML has seen the PI invited to speak at multiple high-level events such as the ERC Conference on Frontier Research in AI (Brussels, October 2018) and TEDx Lugano (September 2019). The PI gave keynote talks at MICCAI Workshops on Graphs in Biomedical Image Analysis (GRAIL) and Shape in Medical Imaging (ShapeMI), Graph Signal Processing Workshop (GSP), lnternational Workshop on Differential Geometry in Computer Vision and Machine Learning (DiffCVML), lnternational Conference on 3D Vision (3DV), ECCV Workshop Geometry Meets Deep Learning (GMDL), ICLR Workshop on Representation Learning on Graphs and Manifolds, International Conference on Medical Imaging with Deep Learning (MIDL), and International Conference on Learning, Optimization and Data (LOD). Invited talks and seminars included top-notch institutions such as the Kavli Institute for Theoretical Physics (UCSB), Broad Institute (Harvard/MIT), Institute for Advanced Study in Princeton, Alan Turing Institute for Data Science, Royal Society, leading universities such as MIT, Harvard, Princeton, Yale, Oxford, and Cambridge, and top tech companies such as Google, DeepMind, Facebook, Twitter, Samsung, and Intel. The PI also gave invited talks at summer schools at SGP (France, 2018), MISS (Italy, 2018), MLSS (Moscow, 2019, Tuebingen 2020, Taipei 2021), AI4Health (Paris 2020), Winter School on Machine Learning (Pokhara 2019), GeoCow (Switzerland, 2020). The PI was a visitor at the IPAM (UCLA) and Institute for Advanced Study (Princeton). The PI organized tutorials on geometric deep learning at NIPS 2017 (extremely popular, with nearly 3000 participants), CVPR, SIGGRAPH, and EUROGRAPHICS. The PI has organized several conference in the domain of Geometric ML, including the series of conferences Geometry Meets Deep Learning (GMDL), ELLIS Workshop on Geometric DL, NeurIPS Workshop on Graph Representation Learning, IPAM Workshop on Novel Deep Learning Techniques (with Yann LeCun et al.), and held various positions as area chair (3DV 2017, ICCV 2017, NeurIPS 2020, 2021, ICML 2021), and program committee member in numerous events. 4. Organized conferencesICML Workshop on Graph Representation Learning, 2020 (virtual).ELLIS Workshop on Geometric and Relational Deep Learning, 2020 (virtual).NeurIPS Workshop on Graph Representation Learning, Vancouver, Canada, 2019.Workshop Geometry Meets Deep Learning (GMDL), Seoul, Korea, 2019.ELLIS Workshop on Geometric Deep Learning, San Sebastian, Spain, 2019.Workshop on New Deep Learning Techniques, Institute of Pure and Applied Mathematics (IPAM), UCLA, Los Angeles, USA, 2018.Workshop on Image-based Modeling of Articulated and Deformable Objects, Venice, Italy, 2017.Workshop Geometry Meets Deep Learning (GMDL), Venice, Italy, 2017.Workshop on Deep Learning and Geometry, Kos, Greece, 2 September 2017.5. Selected PublicationsM. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, P. Vandergheynst, Geometric deep learning: going beyond Euclidean data. IEEE Signal Processing Magazine 34(4):18-42, 2017.F. Monti, D. Boscaini, J. Masci, E. Rodolà, J. Svoboda, M. M. Bronstein, Geometric deep learning on graphs and manifolds using mixture model CNNs. CVPR 2017. OralF. Monti, M. M. Bronstein, X. Bresson, Geometric matrix completion with recurrent multi-graph neural networks. NIPS 2017.Z. Lähner, M. Vestner, A. Boyarski, O. Litany, R. Slossberg, T. Remez, E. Rodolà, A. M. Bronstein, M. M. Bronstein, R. Kimmel, D. Cremers, Efficient deformable shape correspondence via kernel matching. 3DV 2017. O. Litany, T. Remez, E. Rodolà, A. M. Bronstein, M. M. Bronstein, Deep functional maps: Structured prediction for dense shape correspondence. ICCV 2017.A. Gehre, M. M. Bronstein, L. Kobbelt, J. Solomon, Interactive curve constrained functional maps, Computer Graphics Forum 37(5):1-12, 2018.L. Wang, A. Gehre, M. M. Bronstein, J. Solomon, Kernel functional maps. Computer Graphics Forum 37(5):27-36, 2018.D. Nogneng, S. Melzi, E. Rodolà, U. Castellani, M. M. Bronstein, M. Ovsjanikov, Improved Functional Mappings via Product Preservation. Computer Graphics Forum 37(2):179-190, 2018. F. Monti, K. Otness, M. M. Bronstein, MOTIFNET: A Motif-Based Graph Convolutional Network for Directed Graphs. DSW 2018.N. Choma, F. Monti, L. Gerhardt, T. Palczewski, Z. Ronaghi, Prabhat, W. Bhimji, M. M. Bronstein, S. R. Klein, J. Bruna, Graph Neural Networks for IceCube Signal Classification. ICMLA 2018. Best paper awardE. Rodolà, Z. Lähner, A. M. Bronstein, M. M. Bronstein, J. Solomon, Functional maps on product manifolds, Computer Graphics Forum 38(1):678-689, 2019. L. Cosmo, M. Panine, A. Rampini, M. Ovsjanikov, M. M. Bronstein, E. Rodolà, Isospectralization, or how to hear shape, style, and correspondence. CVPR 2019. L. Cosmo, S. Melzi, R. Spezialetti, M. M. Bronstein, L. Di Stefano, F. Tombari, E. Rodolà, GFrames: Gradient-Based Local Reference Frame for 3D Shape Matching. CVPR 2019. OralR. Levie, F. Monti, X. Bresson, M. M. Bronstein, CayleyNets: Graph Convolutional Neural Networks With Complex Rational Spectral Filters. IEEE Trans. Signal Processing 67(1):97-109, 2019J. Svoboda, J. Masci, F. Monti, M. M. Bronstein, L. Guibas, PeerNets: Exploiting Peer Wisdom Against Adversarial Attacks, ICLR 2019. Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, J. M. Solomon, Dynamic graph CNN for learning on point clouds, TOG 38(5):1-12, 2019. D. Kulon, H. Wang, R. A. Güler, M. M. Bronstein, S. Zafeiriou, Single Image 3D Hand Reconstruction with Mesh Convolutions, BMVC 2019. G. Bouritsas, S. Bokhnyak, S. Ploumpis, M. M. Bronstein, S. Zafeiriou, Neural 3D Morphable Models: Spiral Convolutional Networks for 3D Shape Representation Learning and Generation. ICCV 2019. K. Veselkov, G. Gonzalez, S. Aljifri, D. Galea, R. Mirnezami, J. Youssef, M. M. Bronstein, I. Laponogov HyperFoods: Machine intelligent mapping of cancer-beating molecules in foods, Nature Scientific Reports 9:9237, 2019. P. Gainza, F. Sverrisson, F. Monti, E. Rodolà, D. Boscaini, M. M. Bronstein, B. E. Correia, Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning, Nature Methods 17:184–192, 2020. Cover paperL Cosmo, A Kazi, SA Ahmadi, N Navab, M Bronstein, Latent-graph learning for disease prediction, MICCAI 2020. B Croquet et al., Unsupervised Diffeomorphic Surface Registration and Non-linear Modelling, MICCAI 2021. I Laponogov et al., Network machine learning maps phytochemically rich “Hyperfoods” to fight COVID-19, Human genomics 15 (1):1-11, 2021.B. Chamberlain et a., GRAND: Graph Neural Diffusion, ICML 2021. MM Bronstein, J Bruna, T Cohen, P Veličković, Geometric deep learning: Grids, groups, graphs, geodesics, and gauges arXiv:2104.13478 2021 (to appear as a book in MIT Press)6. Major AwardsRoyal Academy of Engineering Silver Medal, 2020Fellow, ELLIS 2019Royal Society Wolfson Research Merit Award, 2018Dalle Molle Foundation Prize, 2018 (for research on fake news)ICMLA Best paper award, 2018 (for research on neutrino detection)Fellow, IEEE, 2018 (for contributions to acquisition, processing, and analysis of geometric data)Fellow, IAPR (for outstanding contributions to 3D data acquisition, processing, representation and analysis), 2018Radcliffe fellowship, Harvard University, 2017Rudolf Diesel industrial fellowship, TU Munich, 2017 Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far) Our group was among the first to bridge geometric and machine learning models, pioneering and spearheading the nascent field of "Geometric Machine Learning" (a term coined by our group). Our geometric deep learning algorithms have been recently used for neutrino interaction classification (a collaboration with IceCube, NYU and UC Berkeley) and fake news detection on social media (our startup Fabula AI acquired by Twitter in 2019). Our group collaborates with partners from world’s leading institutions such as Stanford, MIT, UC Berkeley, NYU, Oxford, Cambridge, EPFL, Ecole Polytechnique, TUM, and Technion, as well as top industrial labs at Google, Facebook, Intel, and Twitter. Graph neural networks are already used in the industry by companies such as Pinterest, Twitter, Google, and Alibaba. Geometric deep learning is already proving a groundbreaking tool in numerous applications, including drug and material design and discovery, drug repositioning, physical sciences, and computational social sciences. Our group has produced groundbreaking results in the detection of fake news on social media from their spreading patterns using geometric deep learning. An extremely promising direction is computational biology, where our joint work with Prof. Bruno Correia (EPFL) shows the possibility to solve notoriously hard problems in protein science, such as data-driven de novo design protein binders. This research could potentially lead to significantly and faster development of new generations of protein-based drugs (biologics) and therapies against oncological diseases. Research by an MIT group used graph deep learning as virtual screening tool for drug candidates to discover a new class of antibiotics. We believe that in the next few years Geometric ML methods could potentially revolutionize the field.
