The FoTran project started with a successful kick-off workshop in September 2018 that attracted over 70 participants and featured international experts in the field of natural language processing. The initial phase was devoted to the development of state-of-the-art neural machine translation (NMT) systems and models for natural language inference (NLI). We proposed our own architecture for what we call “hierarchical refinement encoders” of natural sentences for the NLI task (Talman et al., 2019) and we developed a novel model for multilingual machine translation based on a shared intermediate layer that learns language-agnostic meaning representations (Vázquez et al., 2020). The latter model is called the “attention-bridge model” as it connects independent source language encoders with similarly independent target language decoders via so-called attention links, which summarize semantic information from the input and provide contextualised meaning representations to the output text generator.
We have tested the model in various settings and a detailed analysis of the model’s behaviour is available in Vázquez et al. (2020) showing that knowledge can be transferred from one language to another leading to improved translation quality and the possibility to translate between languages without explicit training data. Furthermore, our experiments support the claim that multilingual setups lead to improved abstractions, which becomes visible in semantic probing tasks and downstream applications that require natural language understanding. We carefully studied the learning dynamics of neural translation models and compared the behavior with language models and different training objectives.
Another line of research that we stress is the interpretation of neural models and the analyses of their generality in terms of their cross-domain and cross-task applications. Neural models are non-transparent black boxes and notoriously difficult to understand. We shed some light on the behaviour of neural translation models and published our work on the linguistic interpretation of model patterns in several publications (Raganato and Tiedemann, 2018; Vázquez et al., 2020). Furthermore, we performed important experiments on the lack of generality in state-of-the-art approaches to NLI showing that such models may fail across domains (Talman and Chatzikyriakidis, 2019).
In the final period we worked on scaling up our modular translation model and looked into the application of multilingual sentence representations in downstream applications. A software framework has been released and widely disseminated (Mickus et al., 2024). We studied the parameters of neural models and how they can be optimised and interpreted. Furthermore, we proposed a methodology for exploring the lexical semantic knowledge encoded in multilingual language models that can be used to build representations of abstract semantic properties with relevance for downstream applications (Garí Soler and Apidianaki, 2021).
We concluded the project with two international workshops, one closing symposium with invited speakers organised in Helsinki and one international workshop on modular and open multilingual NLP (MOOMIN) co-located with EACL 2024.
