Skip to main content

Found in Translation – Natural Language Understanding with Cross-Lingual Grounding

Periodic Reporting for period 1 - FoTran (Found in Translation – Natural Language Understanding with Cross-Lingual Grounding)

Reporting period: 2018-09-01 to 2020-02-29

The goal of the project is to develop models for natural language understanding trained on implicit information given by large collections of human translations. We apply massively parallel data sets of over a thousand languages to acquire meaning representations that can be used for reasoning with natural languages and for multilingual neural machine translation.

Natural language understanding is the “holy grail” of computational linguistics and a long-term goal in research on artificial intelligence. Understanding human communication is difficult due to the various ambiguities in natural languages and the wide range of contextual dependencies required to resolve them. Discovering the semantics behind language input is necessary for proper interpretation in interactive tools, which requires an abstraction from language-specific forms to language-independent meaning representations.

A guiding principle in our research is what we call “cross-lingual grounding” and the effect of resolving ambiguities through translation. The beauty of that idea is the use of naturally occurring data, i.e. human translations, instead of artificially created resources and costly manual annotations in our machine learning approach. The general framework is based on deep learning and neural machine translation and the hypothesis is that training on increasing amounts of linguistically diverse data improves the abstractions found by the model.

The overall objectives of the project are

* the development and implementation of models that can learn sentence-level semantics from massively parallel data sets with over a thousand languages
* the analyses and interpretation of the meaning representations that are learned
* the application of emerging representations as an “interlingua” in machine translation and for advanced reasoning with natural languages

Besides the scientific interest, the project objectives also have a strong societal impact. Language technology already plays a crucial role in human communication in the digital world. Improving the ability to properly understand language input will influence the capabilities of interactive systems and tools. The potentials are enormous. Short term goals include the development of accurate translation services for many more languages with better abstraction, domain knowledge and context-awareness. Long term goals include the development of complex interactive and intelligent machines with human-like language interfaces and deeper world knowledge.
The FoTran project started with a successful kick-off workshop in September 2018 that attracted over 70 participants and featured international experts in the field of natural language processing. The initial phase was devoted to the development of state-of-the-art neural machine translation (NMT) and models for natural language inference (NLI). We proposed our own architecture for what we call “hierarchical refinement encoders” of natural sentences for the NLI task [4] and developed a novel model for multilingual machine translation based on a shared intermediate layer that learns language-agnostic meaning representation [5]. The latter model is called the “attention-bridge model” as it connects independent source language encoders with similarly independent target language decoders via so-called attention links that summarize semantic information from the input and provide contextualised meaning representations to the output text generator.

We have tested the model in various settings and a detailed analysis of the model’s behaviour is available in various scientific publications coming from our team [2,5]. The results show that knowledge can be transferred from one language to another leading to improved translation quality and the possibility to translate between languages for which no explicit training data is available. Furthermore, our experiments support the claim that multilingual setups lead to improved abstractions, which becomes visible in semantic probing tasks and downstream applications that require natural language understanding.

Another line of research that we stress is the interpretation of neural models and the analyses of their generality in terms of their cross-domain and cross-task applications. Neural models are non-transparent black boxes and notoriously difficult to understand. We shed some light on the behaviour of neural translation models and published our work on the linguistic interpretation of model patterns in several publications [1,2,5]. Furthermore, we performed important experiments on the lack of generality in state-of-the-art approaches to NLI showing that such models may fail across domains [4].

Currently, we work on multimodal extensions of our attention-bridge model and look into the application of multilingual sentence representations in downstream applications. We also further study the parameters of neural models and how they can be optimised and interpreted. Our findings will come out soon in new scientific publications.

[1] Alessandro Raganato and Jörg Tiedemann. An Analysis of Encoder Representations in Transformer-Based Machine Translation. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. 2018. DOI: 10.18653/v1/W18-5431
[2] Alessandro Raganato, Raúl Vázquez, Mathias Creutz and Jörg Tiedemann. An Evaluation of Language-Agnostic Inner-Attention-Based Representations in Machine Translation. Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP). 2019. DOI: 10.18653/v1/W19-4304
[3] Aarne Talman. and Stergios Chatzikyriakidis: Testing the Generalization Power of Neural Network Models Across NLI Benchmarks. Proceedings of the Workshop BlackboxNLP on Analyzing and Interpreting Neural Networks for NLP at ACL, p. 85-94., 2019
[4] Aarne Talman, Anssi Yli-Jyrä and Jörg Tiedemann: Sentence Embeddings in NLI with Iterative Refinement Encoders, In Natural Language Engineering. 25, 4, p. 467-482, 2019
Raúl Vázquez, Alessandro Raganato, Jörg Tiedemann and Mathias Creutz. Multilingual NMT with a language-independent attention bridge. Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP). 2019. DOI: 10.18653/v1/W19-4305
Our attention-bridge model implements a novel, scalable architecture that can easily be extended by additional languages, which well supports our goals to dramatically increase language coverage of multilingual translation models. It produces a language-agnostic meaning representation that can directly be applied to other downstream tasks and the architecture provides a flexible framework that can further be augmented with multimodal data such as spoken language or even images. This leads to grounded semantic representations that combine evidence coming from various languages and their connection to audio-visual features. Furthermore, our efforts in explainability and interpretation of neural models increase the ability to understand black-box systems and the linguistic information they pick up from raw data. In the future, we expect further insights coming out of this research and the development of efficient and scalable architectures for multilingual language and translation modeling.
project logotype
project setup
project goals