We proposed the following novel approaches to advance the state of the art in multimodal machine translation, image captioning and their evaluation:
1) Multimodal machine translation: these include statistical approaches that combine captioning and translation models, where the output of captioning models is used to reinforce the statistics of candidate translations (Lala et al., 2017); approaches to re-rank the output of statistical models (Shah et al., 2016) or neural models according to how well they perform in the task of disambiguating ambiguous words, as judged by multimodal word sense translation models (Lala et al., 2018); approaches to learn structured visual information, where this information is used to condition the neural text decoder (Madhyastha et al., 2017) or given to the model for joint learning of how to translate and align regions in images and words in the source sentence (Specia et al., 2019-toAppear); neural approaches using acoustic keys representing domain and speaker information (Deena et al., 2017a, 2017b), approaches using deliberation neural networks to improve translation using a second decoder enhanced with visual information (Ive et al., 2019-toAppear);
2) Image captioning: this is a task closely related to machine translation where the language generation component is conditioned on the image only, instead of on the image and source text. We address this task to explore datasets available for one language, which are much larger than those for machine translation. Our novel approach focuses on using structured object information (presence of objects, their frequency and position) rather than abstract, dense representations, the common practice in the field (Wang et al., 2018);
3) Multimodal lexical translation: instead of translating the whole sentence, lexical translation focuses on ambiguous words, which are arguably harder to translate with text-only models. The approaches we propose treat the problem using recurrent neural networks for tagging or classification, where the visual information is explored through attention mechanisms. The input for such models is a word in its context, the corresponding image (or the region in the image corresponding to the ambiguous word), and the output is the translation of the word in a target language (Lala et al., 2019);
4) Visual content-informed evaluation metric: we propose a novel image-aware metric, VIFIDEL, for the tasks of evaluating image description generation and multimodal translation systems. It estimates the faithfulness of the generated caption with respect to the content of the actual image, based on the semantic similarity between explicit image information and words in the description/translation (Madhyastha et al., 2019);
5) Phrase localization in images: this task is important for aligning words and objects for learning multimodal translation models from structured information. We proposed the first fully unsupervised method for this problem, where no training data or training procedure is required. It uses automatic detections objects, scenes and colours in images, and explores different approaches to measure semantic similarity between the categories of detected visual elements and words in phrases (Wang and Specia, 2019);
6) Simultaneous multimodal translation: we proposed the first approach to simultaneous translation where visual context can be used to anticipate missing textual context, and show that translation quality can be significantly improved in the presence of images or video frames (Caglayan et al., 2020; Ive et al., 2021; Haralampieva et al., 2022-to-appear; Wang et al., 2022-to-appear).
