Graphs without Labels: Multimodal Structure Learning without Human Supervision

Projektbeschreibung

Semantische Struktur aus multimodalen Daten ohne menschliche Annotation lernen

Beim multimodalen Lernen werden Modelle mit Daten aus verschiedenen Modalitäten wie Videos mit visuellen und akustischen Komponenten oder Dokumenten mit Text und Bild trainiert. Das Verfahren beruht auf paarweisen Daten, wie Bild-Text-Paaren, um Deep-Learning-Modelle zu trainieren, damit sie ohne menschliche Überwachung solidere Repräsentationen erlernen. Finanziert über den Europäischen Forschungsrat wird im Projekt GraViLa vorgeschlagen, dass multimodale Modelle effektiv modalitätsübergreifende semantische Einheiten erfassen können und besonders nützlich sind, um Sammlungen vernetzter Modalitäten und Themen zu analysieren, die häufig in multimodalen Dokumenten vorkommen. Das wird im Projekt erreicht, indem semantische Strukturen aus multimodalen Daten über multimodales und selbstüberwachtes Lernen erlernt werden, sodass keine menschlichen Annotationen notwendig sind. Die Daten werden dann als Graph dargestellt, um die Verarbeitung und Deutung umfangreicher Daten zu erleichtern.

Ziel

Multimodal learning focuses on training models with data in more than one modality, such as videos capturing visual and audio information or documents containing image and text. Current approaches use such data to train large-scale deep learning models without human supervision by sampling pair-wise data e.g. an image-text pair from a website and train the network e.g. to identify matching vs. not matching pairs to learn better representations.
We argue that multimodal learning can do more: by combining information from different sources, multimodal models capture cross-modal semantic entities, and as most multimodal documents are a collection of connected modalities and topics, multimodal models should allow us to capture the inherent high-level topology of such data. The goal of the following project is to learn semantic structures from multimodal data to capture long-range concepts and relations in multimodal data via multimodal and self-supervision learning without human annotation. We will represent this information in form of a graph, considering latent semantic concepts as nodes and their connectivity as edges. Based on this structure, we will extend current unimodal approaches to capture and process data from different modalities in a single structure. Finally, we will explore the challenges and opportunities of the proposed idea with respect to their impact on two main challenges in machine learning: data-efficient learning and fairness in label-free learning.
By bridging the gap between those two parallel trends, multimodal supervision and graph-based representations, we combine their strengths of generating and processing topological data, which will not only allow to build new applications and tools but also opens new ways of processing and understanding large-scale data that are out-of-reach at the moment.