Multi-modal Context Modelling for Machine Translation

Información del proyecto

MultiMT

Identificador del acuerdo de subvención: 678017

DOI

10.3030/678017

Proyecto cerrado

Fecha de la firma de la CE 10 Marzo 2016

Fecha de inicio 1 Julio 2016

Fecha de finalización 31 Diciembre 2021

Financiado con arreglo a

• EXCELLENT SCIENCE - European Research Council (ERC)

Coste total

€ 1 493 771,00

Aportación de la UE

€ 1 493 771,00

Coordinado por

IMPERIAL COLLEGE OF SCIENCE TECHNOLOGY AND MEDICINE
United Kingdom

CORDIS proporciona enlaces a los documentos públicos y las publicaciones de los proyectos de los programas marco HORIZONTE.

Los enlaces a los documentos y las publicaciones de los proyectos del Séptimo Programa Marco, así como los enlaces a algunos tipos de resultados específicos, como conjuntos de datos y «software», se obtienen dinámicamente de OpenAIRE .

Publicaciones

Conference proceedings (37)

Phrase Localization Without Paired Training Examples

Autores: Josiah Wang; Lucia Specia
Publicado en: ICCV, 2019
Editor: IEEE
DOI: 10.1109/iccv.2019.00476

Distilling Translations with Visual Awareness

Autores: Julia Ive; Pranava Madhyastha; Lucia Specia
Publicado en: ACL, 2019
Editor: ACL
DOI: 10.18653/v1/p19-1653

How2: A large-scale dataset for multimodal language understanding

Autores: Sanabria, Ramon; Caglayan, Ozan; Palaskar, Shruti; Elliott, Desmond; Barrault, Loïc; Specia, Lucia; Metze, Florian
Publicado en: Visually Grounded Interaction and Language, Neurips, 2018
Editor: Neurips

Findings of the Third Shared Task on Multimodal Machine Translation

Autores: Loïc Barrault; Fethi Bougares; Lucia Specia; Chiraag Lala; Desmond Elliott; Stella Frank
Publicado en: WMT, 2018
Editor: ACL
DOI: 10.18653/v1/w18-6402

Visual Cues and Error Correction for Translation Robustness

Autores: Zhenhao Li, Marek Rei, Lucia Specia
Publicado en: EMNLP, 2021
Editor: ACL
DOI: 10.18653/v1/2021.findings-emnlp.271

Probing the Need for Visual Context in Multimodal Machine Translation

Autores: Ozan Caglayan; Pranava Madhyastha; Lucia Specia; Loïc Barrault
Publicado en: NAACL/HLT, 2019
Editor: ACL
DOI: 10.18653/v1/n19-1422

Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translation

Autores: Julia Ive; Andy Mingren Li; Yishu Miao; Ozan Caglayan; Pranava Madhyastha; Lucia Specia
Publicado en: EACL, 2021
Editor: ACL
DOI: 10.18653/v1/2021.eacl-main.281

Object Counts! Bringing Explicit Detections Back into Image Captioning

Autores: Josiah Wang; Pranava Madhyastha; Lucia Specia
Publicado en: NAACL/HLT, 2018
Editor: ACL
DOI: 10.18653/v1/n18-1198

Exploring Supervised and Unsupervised Rewards in Machine Translation

Autores: Julia Ive; Zixu Wang; Marina Fomicheva; Lucia Specia
Publicado en: EACL, 2021
Editor: ACL
DOI: 10.18653/v1/2021.eacl-main.164

End-to-end Image Captioning Exploits Multimodal Distributional Similarity

Autores: Madhyastha, Pranava; Wang, Josiah; Specia, Lucia
Publicado en: 29th British Machine Vision Conference, 2018
Editor: BVMC

Simultaneous Machine Translation with Visual Context

Autores: Ozan Caglayan; Julia Ive; Veneta Haralampieva; Pranava Madhyastha; Loïc Barrault; Lucia Specia
Publicado en: EMNLP, 2020
Editor: ACL
DOI: 10.18653/v1/2020.emnlp-main.184

Predicting Actions to Help Predict Translations

Autores: Wu, Zixiu; Ive, Julia; Wang, Josiah; Madhyastha, Pranava; Specia, Lucia
Publicado en: Workshop The How2 Challenge: New Tasks for Vision & Language, ICML, 2019
Editor: ICML

VIFIDEL: Evaluating the Visual Fidelity of Image Descriptions

Autores: Pranava Madhyastha; Josiah Wang; Lucia Specia
Publicado en: ACL, 2019
Editor: ACL
DOI: 10.18653/v1/p19-1654

MultiSubs: A Large-scale Multimodal and Multilingual Dataset

Autores: Wang, Josiah; Madhyastha, Pranava; Figueiredo, Josiel; Lala, Chiraag; Specia, Lucia
Publicado en: LREC, 2022
Editor: LREC

Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale

Autores: Ozan Caglayan; Pranava Madhyastha; Lucia Specia
Publicado en: COLING, 2020
Editor: ACL
DOI: 10.18653/v1/2020.coling-main.210

Imperial College London Submission to VATEX Video Captioning Task

Autores: Caglayan, Ozan; Wu, Zixiu; Madhyastha, Pranava; Wang, Josiah; Specia, Lucia
Publicado en: 3rd Workshop on Closing the Loop Between Vision and Language, ICCV 2019, 2019
Editor: ICCV

BERTGen: Multi-task Generation through BERT

Autores: Faidon Mitzalis; Ozan Caglayan; Pranava Madhyastha; Lucia Specia
Publicado en: ACL/IJCNLP, 2021
Editor: ACL
DOI: 10.18653/v1/2021.acl-long.503

Object Counts! Bringing Explicit Detections Back into Image Captioning

Autores: Wang, Josiah; Madhyastha, Pranava; Specia, Lucia
Publicado en: Proceedings of 2018 Conference of the North American Chapter of the Association for Computational Linguistics, Edición 2, 2018
Editor: Computer Vision and Pattern Recognition

Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description

Autores: Elliott, Desmond; Frank, Stella; Barrault, Loïc; Bougares, Fethi; Specia, Lucia
Publicado en: Proceedings of the Second Conference on Machine Translation, 2017, Página(s) 215--233
Editor: Association for Computational Linguistics

Defoiling Foiled Image Captions

Autores: Madhyastha, Pranava; Wang, Josiah; Specia, Lucia
Publicado en: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2018), Edición 1, 2018
Editor: Computer Vision and Pattern Recognition

Sheffield MultiMT: Using Object Posterior Predictions for Multimodal Machine Translation

Autores: Pranava Swaroop Madhyastha, Josiah Wang, and Lucia Specia
Publicado en: Second Conference on Machine Translation, 2017, Página(s) 470–476
Editor: Association for Computational Linguistics

Sheffield submissions for wmt18 multimodal translation shared task

Autores: C. Lala, P. Swaroop Madhyastha, Carolina Scarton, L. Specia
Publicado en: Conference on Machine Translation, 2018
Editor: Association for Computational Linguistics

How2: A Large-scale Dataset for Multimodal Language Understanding

Autores: R. Sanabria, O. Caglayan, S. Palaskar, D. Elliott, L. Barrault, L. Specia, F. Metze
Publicado en: NeurIPS Workshop on Visually Grounded Interaction and Language (ViGIL), 2018
Editor: Neurips

Probing the Tole of Images in Multimodal Machine Translation

Autores: O. Caglayan, L. Barrault, P. Madhyastha, L. Specia
Publicado en: NAACL, 2019
Editor: Association for Computational Linguistics

Findings of the Third Shared Task on Multimodal Machine Translation

Autores: L. Barrault, F. Bougares, L. Specia, C. Lala, D. Elliott, S. Frank
Publicado en: Conference on Machine Translation, 2018
Editor: Association for Computational Linguistics

Semi-supervised Adaptation of RNNLMs by Fine-tuning with Domain-specific Auxiliary Features

Autores: Salil Deena, Raymond W. M. Ng, Pranava Madhyastha, Lucia Specia, and Thomas Hain
Publicado en: INTERSPEECH, 2017, Página(s) 2715–2719
Editor: INTERSPEECH

Multi30K: Multilingual English-German Image Descriptions

Autores: Desmond Elliott, Stella Frank, Khalil Sima'an, and Lucia Specia
Publicado en: 5th Workshop on Vision and Language, 2016, Página(s) 70–74
Editor: Association for Computational Linguistics

Multimodal Lexical Translation

Autores: Chiraag Lala and Lucia Specia
Publicado en: Eleventh International Conference on Language Resources and Evaluation, 2018, Página(s) 3810–3817
Editor: ELRA

Exploring the use of acoustic embeddings in neural machine translation

Autores: Salil Deena, Raymond Ng, Pranava Madhyastha, Lucia Specia, and Thomas Hain
Publicado en: IEEE ASRU Workshop, 2017
Editor: IEEE

SHEF-Multimodal: Grounding Machine Translation on Images

Autores: Kashif Shah, Josiah Wang and Lucia Specia
Publicado en: Conference on Machine Translation, 2016
Editor: Association for Computational Linguistics

A Shared Task on Multimodal Machine Translation and Crosslingual Image Description

Autores: Lucia Specia, Stella Frank, Khalil Sima'an, and Desmond Elliott
Publicado en: First Conference on Machine Translation, 2016, Página(s) 540–550
Editor: Association for Computational Linguistics

Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description

Autores: Desmond Elliott; Stella Frank; Loïc Barrault; Fethi Bougares; Lucia Specia
Publicado en: WMT, 2017
Editor: ACL
DOI: 10.18653/v1/w17-4718

Transformer-based Cascaded Multimodal Speech Translation

Autores: Wu, Zixiu; Caglayan, Ozan; Ive, Julia; Wang, Josiah; Specia, Lucia
Publicado en: 16th International Conference on Spoken Language Translation, 2019
Editor: ACL
DOI: 10.5281/zenodo.3525551

Defoiling Foiled Image Captions

Autores: Pranava Madhyastha; Josiah Wang; Lucia Specia
Publicado en: NAACL/HLT, 2018
Editor: ACL
DOI: 10.18653/v1/n18-2069

Grounded Word Sense Translation

Autores: Chiraag Lala, Pranava Madhyastha, Lucia Specia
Publicado en: 2nd Workshop on Shortcomings in Vision and Language, 2019
Editor: ACL
DOI: 10.18653/v1/w19-1808

Multimodal Quality Estimation for Machine Translation

Autores: Shu Okabe, Frédéric Blain, Lucia Specia
Publicado en: ACL, 2020
Editor: ACL
DOI: 10.18653/v1/2020.acl-main.114

Cross-lingual Visual Pre-training for Multimodal Machine Translation

Autores: Ozan Caglayan; Menekse Kuyu; Mustafa Sercan Amac; Pranava Madhyastha; Erkut Erdem; Aykut Erdem; Lucia Specia
Publicado en: EACL, 2021
Editor: ACL
DOI: 10.18653/v1/2021.eacl-main.112

Peer reviewed articles (7)

MSVD-Turkish: a comprehensive multimodal video dataset for integrated vision and language research in Turkish

Autores: Begum Citamak; Ozan Caglayan; Menekse Kuyu; Erkut Erdem; Aykut Erdem; Pranava Madhyastha; Lucia Specia
Publicado en: Machine Translation, 2021, ISSN 0922-6567
Editor: Kluwer Academic Publishers
DOI: 10.1007/s10590-021-09276-y

Grounded Sequence to Sequence Transduction

Autores: Lucia Specia, Loic Barrault, Ozan Caglayan, Amanda Duarte, Desmond Elliott, Spandana Gella, Nils Holzenberger, Chiraag Lala, Sun Jae Lee, Jindrich Libovicky, Pranava Madhyastha, Florian Metze, Karl Mulligan, Alissa Ostapenko, Shruti Palaskar, Ramon Sanabria, Josiah Wang, and Raman Arora
Publicado en: IEEE Journal of Selected Topics in Signal Processing, 2020, ISSN 1932-4553
Editor: Institute of Electrical and Electronics Engineers
DOI: 10.1109/jstsp.2020.2998415

Unraveling the Contribution of Image Captioning and Neural Machine Translation for Multimodal Machine Translation

Autores: Chiraag Lala, Pranava Madhyastha, Josiah Wang, Lucia Specia
Publicado en: The Prague Bulletin of Mathematical Linguistics, Edición 108/1, 2017, Página(s) 197–208, ISSN 1804-0462
Editor: De Gruyter
DOI: 10.1515/pralin-2017-0020

The role of image representations in vision to language tasks

Autores: PRANAVA MADHYASTHA, JOSIAH WANG, LUCIA SPECIA
Publicado en: Natural Language Engineering, Edición 24/03, 2018, Página(s) 415-439, ISSN 1351-3249
Editor: Cambridge University Press
DOI: 10.1017/s1351324918000116

Assessing multilingual multimodal image description: Studies of native speaker preferences and translator choices

Autores: STELLA FRANK, DESMOND ELLIOTT, LUCIA SPECIA
Publicado en: Natural Language Engineering, Edición 24/03, 2018, Página(s) 393-413, ISSN 1351-3249
Editor: Cambridge University Press
DOI: 10.1017/s1351324918000074

Read, spot and translate

Autores: Lucia Specia; Josiah Wang; Sun Jae Lee; Alissa Ostapenko; Pranava Madhyastha
Publicado en: Machine Translation, 2021, ISSN 0922-6567
Editor: Kluwer Academic Publishers
DOI: 10.1007/s10590-021-09259-z

Multimodal machine translation through visuals and speech

Autores: Umut Sulubacak; Ozan Caglayan; Stig-Arne Grönroos; Aku Rouhe; Desmond Elliott; Lucia Specia; Jörg Tiedemann
Publicado en: Machine Translation, 2020, ISSN 0922-6567
Editor: Kluwer Academic Publishers
DOI: 10.5281/zenodo.3690790